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EXECUTIVE SUMMARY 
TERMS OF REFERENCE 


The NAPLAN Reporting Review was commissioned by the Education Council of COAG and has been 
framed by four Terms of Reference: 
1. Perceptions of NAPLAN and My Schoo! data, including the potential for misinterpretation or 
misuse of data; 
2. How My School and NAPLAN reporting contribute to understanding of student progress and 
achievement; 
3. How schools use achievement data, including NAPLAN, to inform teaching; and 
4. How My School and NAPLAN data are reported to students and parents. 


PROCESS 


This Review began with an analysis of publicly available school-level reporting in comparable English- 
speaking jurisdictions. Jurisdictions were chosen to provide maximum variation in possible 
approaches (Chapter 1). Focus group interviews were conducted with more than 200 school leaders, 
teachers, parents and students in a purposive sample of 10 high-gain schools (Chapter 2). Thirty- 
three public submissions were received, and interviews were conducted with more than 70 
stakeholders representing 23 school system, sector, union and parent stakeholder groups (Chapter 
3). Findings and recommendations are discussed in Chapter 4 and summarised below. 


FINDINGS 

In the list of Findings below, Terms of Reference 2, 3 and 4 are considered before Term of Reference 
1. The purpose of this re-ordering is to provide findings about patterns of use of NAPLAN and My 
School by schools, systems and sectors, parents and students, before returning to broader 
perceptions about whether these patterns of use reflect an appropriate balance between the right 
to high quality information and the possibility of misunderstanding or misuse of data. 


Understanding student progress and achievement 


1. Australian governments and school systems rely on NAPLAN to make judgements about school 
and school system performance, but some have reservations about unintended consequences 
of NAPLAN testing and the publication of NAPLAN data. 


2. School sector and system data analytics platforms are widely used in understanding student 
progress and achievement, but schools do not use My School data displays for this purpose. 


3. NAPLAN provides important but incomplete information on school quality. 


4. Data displays that focus on gain in student achievement were preferred to those that used 
ICSEA-based calculations to compare statistically similar schools. 
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Using achievement data to inform teaching 


5. Schools triangulate NAPLAN data with a wide range of other assessments including large-scale 
nationally normed and standardised tests. 


Reporting NAPLAN data to parents and students 


6. The My School website provides more comprehensive and detailed school-level performance 
data displays than any of the international jurisdictions considered in this Review, but less 
information than is provided in Australian system and sector data analytics platforms. 


7. School-level NAPLAN results are widely but not universally available from sources other than 
My School. 


8. Schools make limited use of school-level NAPLAN data in communication with families and 
prefer timelier and more contextual data when discussing individual student achievement. 


9. Parents generally do not place a high priority on NAPLAN results when choosing a school, but 
many believe that transparency and accountability require publication of these results. 


10. My School and NAPLAN would be strengthened by a public communication program that 
clarified the purposes and proper uses of the data and the website. 


11. Students would benefit from age-appropriate explanations of their NAPLAN results. 


Perceptions of NAPLAN and My School 


12. There is no settled view of the purposes or proper uses of NAPLAN and the My School website. 
Further clarification and communication of these issues would be welcomed. 


13. Many stakeholders were concerned that publishing school-level NAPLAN data had made the 
tests “high stakes”, and that any negative consequences flowed from publication of NAPLAN 
data rather than the collection of data or provision of data to schools and school systems. 


14. Colour-coding of NAPLAN results was regarded as useful by many focus group participants, but 
stakeholders had concerns about the use of current ICSEA comparisons as a basis for the similar 
school calculations that underpin the colour-coding. A technical review of ICSEA would be well 
regarded. 


15. Measures of student achievement and gain may be sufficient information for public 
accountability and transparency purposes. 


16. Some, but not all, of the recent Australian league tables have drawn their NAPLAN data from 
publicly available sources other than My School. 


17. Concerns about the impact of NAPLAN on teaching and learning programs were reported but 
estimates of the severity of this impact varied among stakeholders and data sources. 


18. Concerns about the impact of NAPLAN on wellbeing of teachers and students were reported 
but estimates of the severity of this impact varied among stakeholders and data sources. 
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19. There is widespread interest in the development of on-line, formative assessments based on 
learning progressions. 


20. Some stakeholders advocated for a national accountability system based on sample testing, but 
this is inconsistent with school systems’ and sectors’ current use of population NAPLAN data in 
their data analytics, school review systems and school board reports. 


RECOMMENDATIONS 


1. That the number of NAPLAN displays on My School be reduced. 


2. That the focus of NAPLAN displays on My School should be student gain, not statistically similar 
school comparisons. 


3. Thata technical review of ICSEA be undertaken. 


4. That the national priority initiatives on learning progressions and formative assessment tools be 
pursued, in order to improve the timeliness and diagnostic quality of assessments available to 
schools. 


5. That school systems publish school-level NAPLAN data in ways that reduce the likelihood that 
third-party NAPLAN-based school league tables will be produced. 


6. Thatin order to reduce the risk of misuse of NAPLAN data, clear guidance be provided to 
schools, the public and students about the purposes and proper uses of NAPLAN and My School. 
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INTRODUCTION 


This Review, commissioned by the Education Council of the Council of Australian Governments in 
September 2018, has four Terms of Reference: 
1. Perceptions of NAPLAN and My Schoo! data, including the potential for misinterpretation or 
misuse of data; 
2. How My School and NAPLAN reporting contribute to understanding of student progress and 
achievement; 
3. How schools use achievement data, including NAPLAN, to inform teaching; and 
4. How My School and NAPLAN data are reported to students and parents. 


The Review has undertaken four phases of data collection 


Environmental scan 

The environmental scan explored local and international representations of student achievement 
data. The focus was on school-level summaries of achievement in the primary and lower-secondary 
years, available on public websites. International jurisdictions chosen were those with 
predominately English-language education systems in countries with similar or better national 
student achievement. In countries with many internal education jurisdictions, a maximum variation 
sample was chosen. 


Sample school community consultation with teachers, school leaders, parents and students 
School community consultation was undertaken with school visits late in 2018 and early in 2019. 
Phase 2 involved a purposive sample of 10 schools selected in collaboration with the Australian 
Curriculum Assessment and Reporting Authority. Schools in the sample have at least average 
national NAPLAN achievement and have shown greater growth in achievement than schools with 
similar starting points or statistically similar schools. The sample represents a range of school 
jurisdictions, locations, demographic characteristics and phases of schooling. In each school, face-to- 
face focus group interviews withheld with school leaders, teachers, parents and students. 


Public submissions 

Written submissions were sought over a six-week period beginning in February 2019. An Issues 
Paper was released to guide and structure written submissions to the Review. Thirty-three written 
submissions were received. People and organisations making written submissions are identified in 
Appendix 1. A link to the Issues Paper is provided in Appendix 3. 


Key stakeholder consultation 

Face-to-face individual consultations were conducted with national peak bodies and school system 
and sector authorities during March and April 2019. Twenty-three stakeholder groups were 
consulted. Seventy-three stakeholder representatives attended these meetings, almost all of which 
were face-to-face meetings. Appendix 2 identifies groups attending stakeholder consultation 
meetings. 
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CHAPTER 1: ENVIRONMENTAL SCAN 


MY SCHOOL AND NAPLAN ACHIEVEMENT REPORTING 


MY SCHOOL 


The National Assessment Program — Literacy and Numeracy (NAPLAN) was introduced in 2008, 
replacing a series of separate state and territory assessments. Since 2010, NAPLAN has been the 
responsibility of the Australian Curriculum, Assessment and Reporting Authority (ACARA). With 
regard to assessment, ACARA’s legislative functions are to: 
e develop and administer national assessments; and 
e collect, manage and analyse student assessment data and other data relating to schools and 
comparative school performance; and 
e facilitate information sharing arrangements between Australian government bodies in 
relation to the collection, management and analysis of school data; and 
e publish information relating to school education, including information relating to 
comparative school performance.’ 


In January 2010 ACARA launched the My School website, containing school-by-school summaries of 
2008 and 2009 NAPLAN data in reading, writing, spelling, grammar and punctuation and numeracy 
at Years 3, 5, 7 and 9. The My School website initially provided comparisons with statistically similar 
schools, with measures of achievement gain added in 2011. The initial version of My School also 
provided a range of school demographic information, augmented by Information on school finance 
and capital expenditure in 2011. 


The 2018 version of the My School website provided five school-level analyses of achievement and 
impact. These include: 
e Number: average achievement for each assessment domain, over time; 
e Bands: the percentage of students in each achievement band; 
e Graphs: average scores for each assessment domain, over time; 
e Student gain: average change in results for students who have taken successive NAPLAN 
tests in the same school; and 
e Similar schools: average scores for each assessment domain and each calendar year in 
schools with a similar ICSEA value to the selected school. 


For 2019, an additional display has been developed: 
e = Participation: Student participation in NAPLAN displayed in a table. 


1 https://www.legislation.gov.au/Details/C2008A00136 
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Each of these analyses allows users to make comparisons between statistically similar schools or all 
Australian students. Statistical similarity of schools is calculated using an index constructed for the 
purpose, the Index of Community Socio-Educational Advantage. In addition, the student gain page 
allows comparisons between the selected school and students with the same starting scores. 


The My School website provides colour-coding of NAPLAN results for average scores “substantially 
above”, “above”, “close to”, “below” or “substantially below” schools with similar students or all 
Australian schools. Statistical confidence intervals are also provided, where appropriate, to limit the 


possibility of over-interpretation of small differences in the data. 


SCHOOL SYSTEM NAPLAN DATA PRESENTATIONS 


In addition to the NAPLAN data displays available to the public through the My Schoo! website, 
Australian school systems and sectors have invested substantially in data analytics platforms to 
support their schools. More than a dozen such platforms have been identified.” To take one example 
of these data analytics platforms, the NAPLAN component of Scout provides online, graphics- 
intensive information on school performance, student performance and NAPLAN item-level 
performance.? 


The school performance component of Scout includes the following displays (see Figure 1): 
e NAPLAN scores over time; 
e equity groups report; 
e number and percentage of students in achievement bands over time; 
e percentage of students in Bands versus a statistically similar school group and the whole 
state; 
e percentage of students in the top two bands in reading and numeracy; and 
e student growth in scores and across bands. 


? See the section “Assessment tools and data analytics”, beginning page 78. 
3 https://education.nsw.gov.au/about-us/educational-data/scout/media/documents/aqrg-non-doe-smart-to- 
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Figure 1. Scout school performance displays 
% of students in Bands over Time 
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Student Growth Across Bands report 
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In addition to this school-level data, Scout enables schools to explore individual students’ NAPLAN 
performance across the tests and years, as well as student and cohort performance at the individual 


NAPLAN item level (Figure 2). 


Figure 2. Scout individual student and item displays 
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INTERNATIONAL SCHOOL-LEVEL ACHIEVEMENT REPORTING 


Not all comparable jurisdictions have whole-population testing in the primary and lower-secondary 
years. Some have jurisdiction- or nation-wide assessments but choose not to publish school-level 
data or use sample tests for accountability purposes. Most of the school systems comparable in size 
to the Australian state and national systems do, however, publish some kind of school-level student 
achievement data, often accompanied by value-added, student gain or similar-schools estimates of 
school-level impact on student achievement. 


International comparisons in this Review are focused on countries with predominantly English- 
language medium schools and broadly comparable or better performance. In two of the larger 
federations, the United States and Canada, there are many jurisdictions: 50 US states and ten 
Canadian provinces plus more than 20 territories across both countries. Between the US and 
Canada, both of which manage schools though local school boards and districts, there are tens of 
thousands of local school districts. Some of these are as large as the most populous Australian 
states. In order to show the range of school-level achievement reporting practices across these 
many jurisdictions maximum variation sampling has been used. Jurisdictions described include the 
largest Canadian province, Ontario, three US states (Tennessee, California and Arizona) and largest 
US school district, New York City. 


Beyond North America, school-level achievement reporting practices are described for Singapore, 
Hong Kong and the United Kingdom. 


New Zealand is included in the summary tables, but not the discussion, because it no longer has a 
whole-population national assessment system for students below school leaving age. The National 
Monitoring of Student Achievement program, a set of sample studies involving “several thousand 
children each year on different areas of the curriculum” will replace population testing, but details of 
the program have not yet been released.’ Currently, the Education Review Office publishes the only 
compulsory school-level reporting for primary and lower-secondary schools in New Zealand.° These 
school reports do not include student achievement data. The ERO reviews follow a school self- 
assessment and result in a 3-4 page public report, which provides demographic information, a 
review of progress against priorities and a summary rating on a four-level verbal scale: “Needs 
development”, “Developing”, “Well placed” and “Strong”. The only public reporting of school-level 
achievement data in New Zealand is at the end of secondary schooling, where National Certificate of 
Educational Achievement league tables are published by the New Zealand Herald.® 


* https://education.govt.nz/news/national-standards-removed/ 
2 https://www.ero.govt.nz/review-reports/ 


° https://insights.nzherald.co.nz/article/ncea-table-2018/ 
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SINGAPORE 


In Singapore, The Straits Times produced academic achievement rankings of secondary schools for 
two decades, using Ministry of Education data. The rankings were abolished in 2012 following a 
Ministry of Education decision to broaden the goals of schooling.’ 


School performance information continues to be available on a searchable Ministry of Education 
website.® For secondary schools, this includes Primary School Leaving Examination aggregate scores 
for students entering the school each year in each academic stream (Figure 3). Lower, upper and 
median scores for each stream in the school are reported. From 2021 numerical scores will be 
replaced by band scores on an eight-level achievement scale.° Currently, third party websites such as 
The Asian Parent Singapore produces annual secondary school league tables based on PSLE cut-off 
scores.?° 


Figure 3. Singapore Primary School Leaving Examination intake scores 


PSLE Aggregate Score 
Aggregate range of 2017 Primary 6 students posted to 2018 Secondary 1 


Non-Affiliation Affiliation 
Lower Upper *Median Upper 


Express 211 2 217 None-posted 
Normal Academic 6 75 None-posted 


Normal Technical : None-posted 


HONG KONG 


Hong Kong has a Territory-wide Assessment System with tests at the end of Key Stages 1, 2 and 3 


(equivalent to Years 3, 6 and 9). Territory-wide results are released each year, reporting the 
percentage of students achieving basic competency in Chinese language, English language and 
mathematics.*? School-level TAS results are not released to the public. Access to school-level data is 
restricted and the policy is that “comparisons on the performances among schools should not be 
made’”.’? In Since 2018, in order to make the Key Stage 1 assessments “a low-stakes assessment 
without the need for drilling”, sample rather than whole-population test have been used with no 
student or school names collected in the sampling process.?? 


In the absence of rankings based on school achievement, other rankings have emerged in Hong 
Kong. Book of School, for example, creates rankings based on how competitive entry is to 


7 http://eresources.nlb.gov.sg/infopedia/articles/SIP_512 2005-01-03.html 

8 https://beta.moe.gov.sg/schools/school-finder/ 

? https://www.moe.gov.sg/microsites/psle/PSLE%20Scoring/psle-scoring.html 
10 https://sg.theasianparent.com/psle-results-2018 

11 https://www.bca.hkeaa.edu.hk/web/TSA/en/PriTsaReport.html 

12 https://www.bca.hkeaa.edu.hk/web/TSA/en/PriFaq.html 

13 https://www.info.gov.hk/gia/general/201810/23/P2018102300398.htm 
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kindergartens and primary schools, and on secondary school leaving results and university 
placement. Non-academic rating components include success in sport and music competitions, 
teacher quality and school physical facilities.“* Book of School secondary school rankings for the top 
two schools in 2018 appear in Figure 4. 


Figure 4. Hong Kong: Book of School secondary school rankings 


Academic Sport 15% : Teacher Campus —_scorethis 
40% sched 15% 15% year 


position 
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UNITED STATES OF AMERICA 


The United State of America has a long history of public display of school-level achievement data. 
Under the No Child Left Behind Act of 2001, all US states were required to give basic skills 
assessments to all students annually in grades 3-8. In addition, they were required to display student 
achievement data in annual report cards on schools and school districts.*° 


In 2015, this legislation was replaced by the Every Student Succeeds Act, which continued to require 
universal assessment but left the details of school and district-level reporting to individual states. 
Under these more recent rules, a variety of school-level report cards have been developed by states 
and school districts. 


Some of the report cards are highly stylised and dependent on graphic, letter- or number-grades 
intended to summarise school performance. At the other end of the spectrum are report cards that 
provide detailed and data-intensive displays of achievement, gain and comparisons with statistically 
similar schools. 


In addition to government sanctioned websites, school-level performance data is available on a 
number of third-party websites that rank schools or allocate numerical or letter grades. Typically, 
these use compound indicators, adding scores from multiple tests or combining test scores with 
measures of student growth, equity, diversity or teacher quality. 


The sections below describe state or school district report cards selected to show the range of data 
displays as well as a selection of third-party school-rating websites. 


14 
https://www.bookofschool.com/school/controller/schoolSearch?reporttype=secondaryranking&schoolcatego 
ry=2 

15 https://www2.ed.gov/nclb/accountability/schools/accountability.html 

16 https://www.ed.gov/essa?src=rn 
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Tennessee School Report Card 


The Tennessee Department of Education report card rates each school using summative numerical 
indicators.” Elementary and middle schools have four indicators: academic achievement, student 
academic growth, chronically out of school and progress on English language proficiency. High 
schools add a further two indicators: ready to graduate and graduation rate. Figure 5 provides a 
sample report card for an elementary school. 


Figure 5. Tennessee: elementary school report card 
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View Details View Details 


Further detail is available on the Tennessee Report Card website regarding academic achievement, 
including the proportion of students who met the state’s performance benchmarks and students’ 
academic growth using a value-added calculation. These benchmark and growth performances are 
also disaggregated by specific student race, socio-economic status, language background and 
disabilities groups. The underlying test scores that lead to these numerical indicators and value- 
added calculations are not available through the public website. 


17 https://reportcard.tnk12.gov/ 
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California School Dashboard 


The California School Dashboard” provides a set of performance indicators in English language arts, 
mathematics, chronic absenteeism and suspension (Figure 6). The academic achievement indicators 
are an average of all students’ distance from the expected grade-level standard, showing by how 
much the average student exceeds the standard, or how much improvement the student would 
need to reach the grade-level standard.?? Distance from the expected standard is represented by a 
fire-hazard graphic. Quantitative data accompanying the graphic indicate the number of test scale 
points above or below standard, as well as change from the previous year’s score. The graphics 
indicate how many equity groups are in each quintile of the performance levels. The underlying 
student achievement data are not available on the dashboard website. 


Figure 6. California School Dashboard 
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18 https://www.caschooldashboard.org/ 
19 https://www.cde.ca.gov/ta/ac/cm/acadindcal.asp 
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Arizona School Report Card 


The school report cards produced by the Arizona Department of Education provide data on student 
achievement in every school as well as a letter grade to summarise performance of every school.” 
The achievement data are available in chart or table form for each grade and for English language 
arts (ELA), mathematics and science. Data displayed in each case include the proportion of students 
in each proficiency band, for all students and in each equity group (Figure 7). 


Figure 7. Arizona School Report Card: academic results 
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In addition, the Arizona system calculates a number of “proficiency points” for each school.?! The 
index includes points for achievement, growth and readiness for future learning as well as bonus 
points for the proportion of special education enrolment and high performance in a science 
assessment. Figure 8 provides an example of bonus points for a school with an A letter grade. 


20 https://azreportcards.azed.gov/ 
21 https://azreportcards.azed.gov/static/A-FSummary 
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Figure 8. Arizona School Report Card 
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New York City School Performance dashboard 


The New York City Department of Education publishes a School Performance Dashboard for each of 
its schools.”? The dashboard includes demographic data, scores on a quality review survey, 
percentile rankings in English language arts and mathematics and a measure of impact (given 
student intake characteristics) versus unadjusted performance (Figure 9). A wide range of trend data 
are available and can be compared with city-wide performance and performance of similar students 
in other schools (Figure 10). Multi-year data tables track achievement over time and are colour- 
coded for above and below average performance compared with city-wide and comparable students 
(Figure 11). 


22 https://tools.nycenet.edu/dashboard/ 
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Figure 9. New York City: elementary school report card: impact and performance 
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Figure 10. New York City: elementary school report card: achievement 
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Figure 11. New York City elementary school report card: comparable school comparisons 
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League tables and rankings 


Independent of the school report cards provided by state or school district education authorities in 
the United States, a variety of third-party organisations provide school rankings and league tables 
based on publicly available student achievement data. Three examples are provided below, all 
referring to the school described in Figure 9, an elementary school on the Upper West Side of 
Manhattan. 


SchoolDigger’’, provides state and city numerical rankings and star ratings based on normalised and 
averaged test scores.”* SchoolDigger rates this Manhattan school as Two Stars and ranks it 1,259 
out of 2,395 New York elementary schools (Figure 12). 


Niche”° provides a letter grade report card, school district-wide rankings on quality, diversity and 
teacher quality, proportion of students meeeting state-wide proficiency levels in reading and 
mathematics and (where appropriate) college preparation.*° Niche ranks this Upper West Side 
school in the top third, 721°" out of 2,467 New York City elementary schools. It receives B+ grade 
overall and B+ for academics, A for diversity and A for teacher quality (see Figure 13). An A grade 
places the school in the top 10% and a B+ grade places the school in the top 30%. 


23 https://www.schooldigger.c om/ 

24 https://www.schooldigger.com/aboutranking.aspx 
25 https://www.niche.com/?ref=k12 

26 https://www.niche.com/k12/rankings/methodology/ 
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GreatSchools”’ provides a similar but more methodologically complex summary. In addition to 
achievement scores, GreatSchools considers student growth, within- and between-school equity of 
achievement, college readiness and proportion of secondary students taking advanced 
coureswork.”® GreatSchools rates this school 7/10 overall with a score of 10/10 for student progress 
and 5/10 for equity of acheivement across social groups (see Figure 14). GreatSchools does not 
provide a district-wide ranking for this school. 


Figure 12. SchoolDigger: New York City elementary school 
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27 https://www.greatschools.org/ 
28 https://www.greatschools.org/gk/ratings/ 
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Figure 14: GreatSchools: New York City elementary school 
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CANADA 


School-level academic achievement results are published by some Canadian provincial governments, 
including two of the four large provinces (Ontario7? and Alberta*’) and several of the smaller 
provinces (Prince Edward Island** and New Brunswick”). British Columbia publishes school-level 
grade-promotion, student enrolment and student satisfaction data, but not provincial test data®’. 
Year 3, 4and 6 provincial assessments have been suspended in Nova Scotia.** There are no whole- 
cohort national assessments in Canada. 


In Ontario, the most populous province, the Education Quality and Accountability Office provides a 
searchable repository of school reports for all publicly-funded Ontario schools. Each school report 
includes demographic information, results of a student attitude survey and the percentage of 
students at or above the provincial standard in reading, writing and mathematics. 


Although the web-based report for the Toronto elementary school in Figure 15 includes some 
contextual information and tracks data over time, it does not compare the school with district- or 
province-wide standards or statistically similar schools. More detailed information, including district 
and provincial comparisons, the percentage of students at each level in each assessment and the 
results of a student survey, is available on a comprehensive downloadable PDF report.*° 


The Frazer Institute, an independent think-tank, produces annual report cards for Canadian 
provinces, ranking schools in the province and rating them on a 10-point scale using a set of 
academic indicators drawn from provincial assessment programs.” In recent years, the Frazer 
Institute has produced rankings of elementary and secondary schools in three of the four large 
provinces — British Colombia, Alberta and Ontario. 


2° http://www.eqao.com/en 


30 https://www.cbe.ab.ca/about-us/provincial-tests-and-reports/Pages/2017-2018-results.aspx 


assessment-results 
32 


https://www2.gnb.ca/content/gnb/en/departments/education/k12/content/anglophone_sector/reports on 
achievement/anglophone_north.html 

33 https://studentsuccess.gov.bc.ca/ 

34 https://novascotia.ca/studentassessments/ 


35 


https://eqaoweb.eqao.com/eqaoweborgprofile/Download.aspx?rptType=PBS& Mident=123056&YEAR=2018 


&assessmentType=6&orgType=S&nF=IsjEuDNrlSUVDWNQOzV9MPfbdC5SJR7N4toXwLYcnTNilw=&displayLanguag 
e=E 


36 https://www.fraserinstitute.org/studies/school-report-cards?page=1 
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Figure 15. EQAO: Toronto elementary school 
GRADE 6 CONTEXTUAL INFORMATION: 2017-2018 


School Board Province 
Number of Students 16 444 132 766 
Number of Classes 1276 8 644 
English Language Learners 4% 11% 
Students with Special Education Needs (excluding gifted) 21% 22% 
First Language Learned at Home Was Other Than English 40% 23% 


STUDENTS’ TIME IN CANADA 

Born in Canada 

In Canada Less Than One Year 

In Canada One Year or More But Less Than Three 
In Canada Three Years or More 


GRADE 6 STUDENT ATTITUDES: 2017-2018 
| like to read I like to write | like mathematics 


(Al Most of the time [| Sometimes il Never (] Blank or ambiguous response 


COMBINED ACHIEVEMENT RESULTS FOR 2014-2016, 2015-2017 AND 2016-2018 


Percentage of All Grade 6 Students At or Above the Provincial Standard 


Reading Writing Mathematics 


: : 83% 84% 84% 
76% 68% 74% 
ill TT 


[B 2014-2016" fj 2015-2017" fi 2016-2018 
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The Frazer Institute’s Report Card on Ontario’s Elementary Schools, for example, uses nine academic 
indicators based on the provincial assessment program.*’ These include average levels of 
achievement in reading, writing and mathematics in the Ontario provincial assessment, gender 
differences in performance and percentage of students not meeting the provincial standard. Figure 
16 below provides an example of a Frazer Institute report card on the same Toronto elementary 
school described in Figure 15. The report includes an overall grade, colour coded on a traffic-signal 
system, as well as the average grade level of achievement on a four-point scale. 


Figure 16. Frazer Institute: Toronto elementary school 


2017-18 Rank 2210/3046 
Rank in the most recent five years n/a 


School Information 


Gr 6 enroliment 


ESL (%) 


Special needs (%c) 


Parents’ average income ($) 


Actual rating vs. predicted based on 
parents’ avg. inc. 


Academic Performance 2014 2015 2016 2017 2018 Trend 
Gr 3 avg. level: Reading 2.3 n/a 2.8 nja 2.8 nja 
Gr 3 avg. level: Writing 2.6 nj/a 2.8 n/a 2.8 nj/a 
Gr 3 avg. level: Math 2.4 2.6 2.2 nj/a 
Gr 6 avg. level: Reading ra 3.0 2.8 nj/a 
Gr 6 avg. level: Writing 2.8 3.0 2.9 nja 
Gr 6 avg. level: Math 2.6 2.6 23 nj/a 
Gender gap (level): Reading nj/a n/a 

Gender gap (level): Math nj/a nja 

Tests below standard (%) 38.5 

Tests not written (%) 5.9 

Overall rating out of 10 


37 https://www.fraserinstitute.org/sites/default/files/ontario-elementary-school-rankings-2019-12659.pdf 
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UNITED KINGDOM 


The UK Government publishes school-level student achievement data for all state-maintained 
primary and secondary schools in England. Neither Scotland, Wales nor Northern Ireland publish 
detailed school-level results for primary schools — although some data are available for both 
Scotland and Wales. 


Scotland publishes the proportion of students achieving the Curriculum for Excellence Level relevant 
to their stage of schooling in decile bands for each school and test domain — for example 70%-<80% 
in Primary 4, in English writing.*° Individual student scores are based on teachers’ judgements. The 
broad bands of performance provide some indication of school-level student achievement but are 
not amenable to the production of school league tables. Scottish National Standardised 
Assessments, first introduced in 2017, are not publicly available on a school-by-school basis.°° 


In Wales, publication of school-level data at Foundation, Key Stage 2 and Key Stage 3 has been 
withdrawn from 2017-18. The Welsh Government has made these changes on the grounds that 
publication of data has narrowed curriculum choice, encouraged competition rather than 
collaboration between schools and had an unnecessary impact on the workload of teachers and 
others without benefit for learners.*° The school inspection system will, however continue, with 
public grading of schools according to the level of external support and guidance they need. The 
news site Wales Online provides a searchable colour-coded database using these grades, with 
schools rated green, yellow, orange or red, based on the amount of support they need.” 


The primary school data displays for England are relatively simple, containing bar graphs and 
percentages of students achieving the expected standard in reading, writing and mathematics, the 
proportion achieving at a higher standard, average scores in reading and mathematics, and a 
measure of progress (growth in student score from one Key Stage test to the next) in reading, 
writing and mathematics. In addition, schools are rated and colour-coded for progress in each test 
domain (Figure 17). * 


38 See the publicly searchable website: 


39 https://www.heraldscotland.com/news/15550882.school-league-table-fears-quashed-as-john-swinney- 


40 https://gov.wales/sites/default/files/publications/2019-02/draft-evaluation-and-improvement- 
accountability-arrangements-for-wales. pdf 

41 https://www.walesonline.co.uk/news/education/school-colour-codes-2019-wales-15754205 

“2 https://www.compare-school-performance.service.gov.uk/%20 
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Figure 17. England: Department for Education primary school display 
Progress score in reading, writing and maths © 


Reading Writing 


More score details Qe More score details (2) More score details (2) 


Pupils meeting expected standard in reading, writing Pupils achieving at a higher standard in reading, 
and maths @ writing and maths O 


School School 
Local authority Local authority 


average average 


England average England average 


Average score in reading e Average score in maths e 


School School 


Local authority Local authority 
average average 


England average England average 


80 (lowest) 100 (expected) 120 (highest) 80 (lowest) 100 (expected) 120 (highest) 


Data displays for secondary school achievement have used a variety of measures over the last 25 or 
so years. Initially, only student achievement data were reported (the percentage of students 
achieving five or more good GCSEs). In 2002 a measure of school impact was added. Called “national 
median line value-added”, this was an estimate of how much better students achieved than was 
predicted by their Year 6 test achievement. 


In 2006, following criticism that this failed to take account of student socio-educational 
characteristics, the impact measure was changed to “contextual value-added”. This statistically 
complex process based on multi-level modelling was replaced in 2011, on the basis of criticism that 
it was too hard for the public to understand and that it reinforced low aspirations of disadvantaged 
students. Contextual value-added was replaced by a new measure called “Expected Progress” — a 
measure of gain rather than value-added. Expected progress measured the proportion of students 
who achieve the expected growth, given their Key Stage 2 scores. In 2016, Expected Progress was 
replaced by “Progress 8”, a measure of gain across eight subjects including English and mathematics, 
and the attainment measure was replaced by “Attainment 8”.*? 


Figure 18 shows school level performance data for a secondary school in the north of England. 
Achievement (Attainment 8) is compared with local authority and national averages. Impact 
(Progress 8) is rated as Average and colour-coded amber. 


43 Leckie, G., & Goldstein, H. (2017). The evolution of school league tables in England 1992-2016: Contextual 
value-added, expected progress’ and ‘progress 8’. British Educational Research Journal, 43(2), 193-212. 
https://doi.org/10.1002/berj.3264 
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Because these school-level data are freely available on the Department for Education website they 
can be in arranged in league tables produced by third parties such newspapers. Figure 19 shows a 
simple, searchable league table provided by The Telegraph newspaper. It includes an achievement 
measure (9-4 grade %), an impact measure (Progress 8) and the school grade provided by Ofstead, 
the national inspection agency. 


Figure 18. England: Department for Education secondary school report 


Progress 8 score @ Entering EBacc @ 


More score details (2) 


School 


Local authority 
average 


England average 


Staying in education or entering employment (2) Grade 5 or above in English & maths GCSEs 2] 


School School 
Local authority Local authority 


average average 
England average England average 


Attainment 8 score @ EBacc average point score e 


School . School 


Local authority Local authority 
average average 


England average . England average 


Figure 19. England: The Telegraph secondary school league table 


Top 100 schools for GCSE results, 2018 


School 9-4 grade (%) Ofsted 


Queen Elizabeth's School, Barnet, Barnet 100 ; Outstanding 
Colchester County High School for Girls, Colchester 100 . Outstanding 
Nonsuch High School for Girls, Sutton 100 : Good 


Beaconsfield High School, Beaconsfield 100 ‘ Outstanding 


Kendrick School, Reading 100 : Outstanding 
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SUMMARY 


Across the Australian and international examples canvassed in this paper there is variation on five 
broad dimensions: public availability of data; measures of achievement; measures of impact; 
performance indicators; and availability of third-party websites. 


PUBLIC AVAILABILITY OF DATA 


School-level student achievement data are available on publicly searchable websites in seven of the 
nine international jurisdictions described in this report. Public access is required by law in the United 
States and is available from government websites in Singapore and about half of the Canadian 
provinces. Public access is restricted in Hong Kong. Scotland has only recently introduced national 
standardised tests and does not publish school-level summaries. New Zealand has recently 
withdrawn its national assessment program and Wales has discontinued publication of its national 
assessment data. 


In Australia, school-level student achievement data are available for all schools through the My 
School website. Although the Scout system used in NSW and ACT schools is not publicly available, 
the underlying NAPLAN data are available through My School. The pattern of available data is 
summarised in Figure 20, where an orange dot signifies that the data are publicly available in that 
jurisdiction and a black dot indicates that these data are available to schools through school-system 
data analytics platforms (with the NSW Scout system used as an example of what is available 


through jurisdictional systems). 


Figure 20. Availability of school-level student achievement data 
V4 NYC | ON ENG | NZ AU 


1. DATA 
Public website 


MEASURES OF STUDENT ACHIEVEMENT 


Almost all of the jurisdictions considered here publish school-level proportions of students achieving 
state, province or territory benchmarks (Figure 21). England provides average as well as benchmark 

scores and Singapore provides the upper and lower score in each school as well as the median score. 
New Zealand does not have a national assessment program until the end of secondary schooling and 
Hong Kong publishes the territory-wide rather than school-level proportion reaching the benchmark. 


Several jurisdictions publish comparisons of school-level data over time. The California School 
Dashboard and Arizona School Report Card show the increase or decrease from the previous year. 


“4 The contractions in the table are SG (Singapore), HK (Hong Kong), ON (Ontario, Canada), ENG (England), CA 
(California), AZ (Arizona), NYC (New York City), NZ (New Zealand), AU (My School) and NSW (NSW/ACT Scout 
data analytics platform). 
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The Ontario EQAQ data displays show the proportion reaching a similar standard for each of the last 
three years. England provides school progress scores over a three-year interval and compares 
progress scores with local authority and national school progress scores. The most comprehensive 
international display of time series information is the New York City Performance dashboard. It 
shows the proportion of students in each school achieving the benchmark over the last the four 
years, as well as and colour-coded differences from similar schools, the city and the state over four 
years. 


Most of the US jurisdictions highlight differences in performance among equity groups. The 
Tennessee system’s detailed reports provide school-level benchmark scores disaggregated by 
student race, socio-economic status, language background and disability status. The California 
system links fire-hazard colours to the performance of students in 12 equity groups. In Arizona the 
proportion of students at each of four proficiency levels is shown by gender, race, language 
background, socio-economic status and special education status. 


My School’s publication of school-level achievement data is more comprehensive than any of the 
international jurisdictions considered here. It publishes average student achievement in numbers, 
percentages of students in bands, average achievement over a decade, student gain from one 
NAPLAN assessment to the next and compares schools with statistically similar schools using 
numbers and colour codes. My School provides the percentage of Indigenous students, the 
percentage with a language background other than English and the percentage of students in each 
quartile of socio-educational advantage but does not disaggregate achievement by the equity 
categories. The Scout tool available to schools in NSW and the ACT includes all of the data available 
in as well as a more comprehensive graphic and search capacity and links to individual student 
achievement. 


Figure 21. School-level measures of student achievement 
SG HK TN co7.\ rV4 homme) | = \ (Ca 74 | AU NSW 


2. ACHIEVEMENT 


Average e e e e 
Band / benchmark e e e e e e e e e 
Over time e e e e e e e 
Subgroups e e e ® e 


MEASURES OF IMPACT 


Because there is a social gradient of achievement in every jurisdiction, with achievement varying by 
student ethnicity, language background and socio-economic status, school systems have long been 
interested in measures of impact that account for this social gradient. Three broad procedures for 
measuring the impact of schools on student achievement may be distinguished: value added, 
student gain and statistically similar groups (Figure 22). 
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Figure 22. School-level measures of impact 


NYC | ON ENG | NZ AU 


3. IMPACT 
Value-added e 
Gain e e e 
Similar schools / e e 
similar students 


Value-added measures use statistical procedures that attempt to separate out or hold constant the 
impact of demographics on achievement. Tennessee has been publishing value-added scores for 20 
years” and is the only jurisdiction among those considered here using value-added calculations to 
measure impact. The current Tennessee School Report Card includes a value-added score on a 0-4 
scale called Student Academic Growth. 


Gain scores are published in Arizona and England, where they have recently replaced value-added 
scores. The gain measure for primary schools in England is based on the proportion of students 
achieving at a higher standard than expected from one Key Stage assessment to the next. Progress 8, 
the secondary school measure, computes gain across eight subjects compared with students with 
similar starting points in Key Stage 2 assessments. Arizona uses a points-based indicator of growth 
from year-to-year in test scores as well as growth of students towards annual achievement targets. 


Similar-student comparisons are made among the 1,700 public elementary schools in the New York 
City public school system. The procedure for identifying the comparison group is based on matching 
each student in a school with the 50 most similar students in terms of prior test results and 
demographics.*° These students form a system-wide comparison group, which are then used to 
calculate a school comparison group. Each school is then compared with its student comparison 
group over time and using colour-coding to show whether average proficiency levels are far above, 
above, below or far below the expected score. 


My School (and Scout) uses two of these three impact procedures: gain and similar schools. Student 
Gain shows the average change in results for students who have taken consecutive NAPLAN tests in 
the same school. This is a statistically simpler procedure than those used in England and Arizona, 
both of which have procedures for considering gain against expected levels of achievement. 
Statistically similar school comparisons on the My School website are calculated differently than the 
NYC comparison schools, involving comparisons with a group of 60 Australian schools with the 
closest ICSEA score rather than aggregating up from individual student previous scores and 
demographic characteristics. 


45 https://tvaas.sas.com/welcome.html?as=c 
46 https://infohub.nyced.org/docs/default-source/default-document- 
library/201617educatorguideems11152017.pdf?sfvrsn=90000a1b 4 
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INDICATORS OF ACHIEVEMENT AND IMPACT 


In addition to reporting scores for achievement and calculations of impact, some jurisdictions 
provide numbers, grades or ratings designed to summarise school performance (Figure 23 and 
Figure 24). 


Figure 23. Indicators of achievement and impact 


NYC | ON = (Ca 74 [ew 
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Achievement e e 
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Figure 24. Performance indicator graphics 


English Language Arts 
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The NYC dashboard provides a great deal of information about achievement, over time and 
compared with schools working with similar students. The comparison data are colour-coded to 
assist in interpretation, but there is no overall grade or ranking to summarise school performance. 
The England school reports award a one of five colour-coded grades for progress ranging from red 
for “Well below average” to dark green for “Well above average”. In Tennessee, a point score on a O- 
4 scale summarises the academic achievement and value-added school impact. The California 
dashboard allocates schools to one of five groups in English language arts and in mathematics, using 
a composite performance indicator represented by a fire hazard icon. Arizona awards an overall 
letter grade, based on the number of points awarded for proficiency and growth and acceleration 
readiness. 
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LEAGUE TABLES 


In almost every jurisdiction considered here, irrespective of government decisions about whether or 
not to make school-level performance data readily available to members of the public, third party 
organisations have published league tables on websites (Figure 25). In some cases, jurisdictions 
publish data in ways that makes school comparisons easy. There is ready access to school-level 
achievement data on report cards in every US state as a result of the ESSA Act (2015). In the UK, 
1991-2018 results and ratings for England are available as a downloadable table or searchable index 
from the gov.uk website. Singapore, about half of the Canadian provincial governments and 
Australia provide such data in easily accessible public forms. In New Zealand school league tables are 
published by national assessment authority on the basis of school leaving results*’, but there are no 
whole-population measures for earlier years of school and no league tables for primary and junior 
secondary years. 


Figure 25. League tables 


V4 NYC | ON ENG | NZ AU NSW 


5. LEAGUE TABLES 


In jurisdictions where data are not published, third parties fill the gap. In Canada, the Frazer Institute 
publishes school level academic results and summary ratings for provinces that make data easily 
accessible and those that do not. Hong Kong does not make-school level achievement data available, 
so the third-party ranking system uses proxy indicators such as competitiveness of entry to schools. 


And in jurisdictions that do publish data, third-party websites such as SchoolDigger, GreatSchools 
and Niche produce simplified ranking and rating systems. Using different ranking strategies these 
league tables produce quite different results. The same Upper West Side elementary school that was 
rated about average by SchoolDigger and well above average by Niche (B+) and GreatSchools (7/10). 


Conclusions 


School-level NAPLAN data have been published in Australia for more than a decade. From the 
beginning, consistent with the ACARA Act, both achievement data and similar-schools based 
comparisons of school impact have been available. 


Compared with the data available on school-level achievement in similar whole-cohort assessments 
in other jurisdictions, the material published on the My Schoo! website is comprehensive. All of the 
jurisdictions except New Zealand publish the proportion of students achieving benchmarks or bands 
and a few also publish average scores. No other jurisdiction publishes bands, averages and time 
series data. Most publish data about achievement over time. About half of the jurisdictions also 


47 https://www.nzqa.govt.nz/studying-in-new-zealand/secondary-school-and-ncea/find-information-about-a- 


school/secondary-school-statistics/ 
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publish estimates of the impact of schools on achievement. Two of the jurisdictions described in this 
report use gain measures, one uses value-added and one uses a similar student index. My School is 
the only performance website that uses two estimates of impact: student gain and similar school 
comparisons. 


This diversity of approaches to reporting reflects local circumstances and approaches to school 
accountability, rather than any obvious relationship between transparency and educational 
outcomes. Among jurisdictions with typically higher school performance than Australia, Singapore 
has high levels of transparency and Hong Kong has low levels of transparency. Within nations with 
broadly similar school achievement, such as the United Kingdom, England has high levels of 
transparency and Scotland and wales have low levels of school-level transparency of achievement 
data. 


Although My School is the most comprehensive of the government reporting systems considered in 
this review, it is less comprehensive than systems that have been developed by Australian school 
systems in the decade since My School was launched. 


The use of colour-coding, number or letter grades or graphics to aid interpretation is common 
among the jurisdictions. In many cases these are composite ratings: average, B+ or amber. My 
School uses colour coding to indicate distance from the expected score but does not summarise 
overall performance in a single number, grade or colour. 


Finally, league tables are published almost everywhere. Among the jurisdictions considered in this 
environmental scan, New Zealand is the only exception; there are no primary or lower-secondary 
school league tables but there are league tables based on senior secondary school public 
examinations. Elsewhere, even in jurisdictions that do not make available school-level achievement 
data, third-party league tables have emerged. 
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CHAPTER 2: SCHOOL COMMUNITY CONSULTATION 


OVERVIEW 


The school community consultation phase of this review involved visits to a purposive sample of 
schools. To guide selection of the schools, ACARA provided a list of potential schools generated by 
the following selection method: 


1. Three sets of student gain scores were computed for numeracy and reading for each of the 

year level ranges: 

e School gain; 

e School gain compared with the gain of students with the same starting point; and 
e School gain compared with the gain of schools with similar ICSEAs. 

2. Schools were identified where students’ mean gain exceeded the national average gain by 
more than one standard deviation in all three types of school gain. 

3. To limit random year-to-year fluctuation, schools were eliminated if the percentage tested in 
the two years was lower than 70%, and the number of students tested in the two years was 
fewer than 15. 

4. In addition, a small group of schools was identified on the basis of the largest starting score 
and similar-schools gains for each domain, cohort and ICSEA category. 

5. If noschools ina particular state had been identified by Steps 1-4, a small group of schools 
with the highest starting score and similar school gains was identified. 


Forty-five schools were initially identified by this procedure. A purposive sample of ten schools was 
than selected to represent all states and territories and all school sectors, and a range of school 
ICSEA scores, geographical locations, percentage of Indigenous students, percentage of students 
with language backgrounds other than English, and school sizes. As several schools on the first 
potential school list were unable to participate, an additional group of eight potential replacement 
schools was also identified. 


Three of the schools were chosen for gains in the Year 7 to Year 9 cohort and seven were chosen for 
gains in the Year 3 to Year 5 cohort. Year 5-7 gains were not considered because Year 5 and Year 7 
assessments are often undertaken in separate schools. 


The final sample of ten schools has the following characteristics: 

e Two schools from NSW and Victoria and one from each other state and territory (Figure 
1). 

e Six schools located in major cities, three outer regional and one inner regional (Figure 2). 

e Eight government schools, one Catholic school and one independent school (Figure 3). 

e Schools ranging in size from 156 to 850 students. 

e ICSEA scores ranging between 943 and 1150 with a mean of 1002 (Figure 4). 

e Percentage of students with language backgrounds other than English ranging from 0% 
to 96% (Figure 5). 

e Percentage of Indigenous students ranging from 0% to 21% (Figure 6). 
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Characteristics of the sample schools are summarised below. 


Figure 26: Sample schools, by 


jurisdiction 
States and territories 1150 
1100 
NY 1050 
1000 
CTW 950 
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Figure 28: Sample school 
locations, by category 
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Figure 30: Sample school 
locations, by school sector 
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Figure 27: Sample school ICSEA scores 
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Figure 29: Percentage of Indigenous students in 
sample schools 


Indigenous students 


Figure 31: Percentage of students with language 
backgrounds other than English 


Language background other than English 
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METHOD 


The school community consultation phase of the project was conducted in Term 4, 2018 and Term 1, 
2019. 


In each sample school, four to six separate focus group interviews have been conducted. At the end 
of each focus group, participants completed a short (1-2 page) questionnaire. The questionnaires 
include a set of Likert rating scales and an opportunity to write a free-form qualitative response. 


Across the focus groups, participants were asked to comment on four issues regarding My School 
and NAPLAN: 

1. The extent to which current presentation of NAPLAN data to schools and their communities 
supports their understanding of student progress and achievement. 

2. Perceptions of NAPLAN reporting and My School data and the extent to which they meet 
reasonable public accountability and transparency expectations and requirements, including 
considering any misinterpretation and misuse of information and subsequent consequences. 

3. How teachers and school leaders use NAPLAN and its results and My School data to inform 
teaching practice. 

4. How teachers and school leaders communicate NAPLAN results and My School data to 
students and parents. 


Qualitative field notes were taken during each focus group and supplemented by participants’ write- 
in comments on the survey form. 


Quantitative data from each school were pooled, with separate analyses undertaken for school 
leaders, teachers, students and parents. 


Ten school visits were undertaken: six public primary schools, a Catholic primary, a public middle 
school, a public Kindergarten to Year 12 school and an independent Kindergarten to Year 12 school. 
A total of 208 school community members have participated in focus group interviews: 29 school 
leaders, 51 teachers, 37 parents and 91 students. 
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SCHOOL COMMUNITY VIEWS 


SCHOOL LEADERS 


School leaders participating in the focus groups discussed a number of topics: the purposes served 
by the 2018 My Schoo! NAPLAN data presentations, the use of My School and other alternatives for 
improving learning, the value of particular My School presentations, the use of colour-coding and 
confidence intervals on the graphs and potential for negative uses of NAPLAN data. 


Purposes for using My Schoo! NAPLAN data 


The NAPLAN data available on the My Schoo! website serves a variety of potential purposes: 
accountability, comparison and choice between schools and school improvement. School leaders’ 
responses appear in Figure 32. 


Figure 32: School leaders: Purposes for My Schoo! NAPLAN data presentations 


School leaders: For what purposes are My School NAPLAN data presentations 
useful? 
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There was very strong support for using the My Schoo! NAPLAN data presentations to inform school 
improvement (79% agreed or strongly agreed) and strong support from school leaders for using 
them for accountability for student achievement (62% agreed or strongly agreed). School leaders’ 
comments included: 

“NAPLAN is essential for school improvement and resource allocation.” 


“NAPLAN is all we’ve got that compares us with the rest of the country.” 


“We look at NAPLAN results as a staff. When (the principal) first arrived, teachers were 
defensive and dismissive. Now they find that it informs planning and practice. They find it 
motivating. We have created a culture of whole-school accountability for kids’ reading 
achievement.” 
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School leaders generally agreed that these data presentations were useful in identifying trends and 

gaps in student achievement (72% agreed or strongly agreed), informing teaching practice (66% 

agreed or strongly agreed) and setting targets for students (66% agreed or strongly agreed). 
“NAPLAN is used for whole-school planning. It lets us know whether we have taught things 
and whether children have learned them.” 


“NAPLAN helps us target teaching. We’ve done extra work on spelling after poor spelling 
results.” 


“Data is useful to identify common errors made, highlighting focus areas for the future.” 
“Comparisons with like schools very helpful for setting targets for school improvement 
“We've triangulated NAPLAN with PAT, A-E and what they are doing in class.” 


The data provides a useful moderation or cross-checking tool against our own in-house data 
sets, rather than a driver of school strategic direction. 


School leaders generally did not agree that the My Schoo! NAPLAN data presentations were useful 
for guiding families’ choice of schools (45% agree or strongly agree) or making comparisons between 
schools (55% agree or strongly agree). As one school leader said: 
“The comparison between schools is the least useful, and potentially undermining, use of 
NAPLAN data within a schooling system.” 


/My School and other standardised assessment dashboards 


My School is the authoritative national source of information on school-level student achievement 
data. Internally, schools across the country have access to a variety of other dashboards and data 
visualisation systems which allow more fine-grained student-level data analysis and often combine 
achievement data with other data such as school attendance and parent, teacher and student 
satisfaction surveys. In addition, schools make substantial use of other standardised student 
achievement data. All ten of the sample schools visited have used ACER’s Progressive Achievement 
Tests. School leaders’ views on the relative usefulness of these alternatives are displayed in Figure 
33. 
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Figure 33: School leaders: Understanding student progress and achievement 


School leaders: Which data presentations are useful in understanding student 
progress and achievement? 
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Most school leaders (72%) agreed that the My Schoo! NAPLAN data presentations were useful in 
understanding student progress and achievement and a further 10% strongly agreed. Typically, the 
school sector or school-based dashboard analyses on NAPLAN data were much more strongly 
supported (28% agreed and 62% strongly agreed). All school leaders agreed (41%) or strongly agreed 
(59%) that other standardised assessment data presentations such as PAT-R and PAT Maths were 
useful in understanding student progress and achievement. 
“OneSchool has the most useful data representations for use at a school level as we use them 
to look deeply at the data to inform teaching practice, identify trends (beyond NMS, MSS and 
Upper two bands) in regards to curriculum areas and learning needs of students. The 
triangulation of this data in OneSchool to A-E reporting is useful in data discussions to 
determine additional areas of learning to clarify teaching not identified/assessed in NAPLAN 
e.g. justifying opinions in analytical ways.” 


“We are fortunate to work at a school where data is seen in context and used along with 
many other resources to inform our teaching. Our job is stressful enough and at other schools 
would potentially be very stressful.” 


“We use the RAAD breakdown by question, looking at kids who got each one correct. We 
looked at that stuff in staff meetings. Looking for what the kids are missing. We are focusing 
on writing this year.” 


“The use of Scout as a tool to assess data and to analyse student performance, in particular 
growth, is useful.” 


“D of E school improvement dashboard is useful for tracking and monitoring student progress 
and identifying strategic directions — whole site improvement planning.” 


“Target setting is based on PAT R and M, not NAPLAN.” 


“We don’t rely heavily on NAPLAN. It usually backs up what we have seen. We use NAPLAN to 
triangulate with PAT and school tests.” 


“We use PAT for item level analysis, grouping, and planning curriculum emphasis.” 
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Specific My Schoo! NAPLAN pages 


Most school leaders regarded all of My Schoo! NAPLAN data pages as useful (see Figure 3). More 
than 80% agreed or strongly agreed that the Numbers, Bands, Graphs and Student Gain pages were 
useful for this purpose (Figure 34). The Bands, Graphs and Student Gain pages were the most 
strongly supported (Bands 31% strongly agree; Graphs 31% strongly agree; Student Gain 52% 
strongly agree). There was less support for the Similar Schools display, with 45% agreeing that it was 
useful and a further 24% strongly agreeing that it was useful. 


Figure 34: School leaders: My School data presentations 
School leaders: Which My School data presentations are useful in 
understanding student progress and achievement? 
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-Colour-coding and representation of measurement uncertainty 


Colour-coding is used on the My Schoo! Numbers page to compare the selected school with schools 
with similar students, or all Australian students. Average student achievement in the selected school 
is colour coded green for “substantially above”, light green “above”, white for “close to”, pink for 
“below” or red for “substantially below” scores in schools with similar students, or all Australian 
students. Similar colour-coding is used on the Graphs page and the Similar Schools page. 


The measurement uncertainty contained in every estimate of average or median achievement is 
shown by a vertical line above and below the average to represent the margin of error at 90% 
confidence intervals for the selected school and schools with similar students. These confidence 
interval ‘whiskers’ appear on both the Graphs and Student Gain pages. 


School leaders were also asked to comment on the use of colour-coding and the representation of 
measurement uncertainty on the graphs (see Figure 35). 
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Figure 35: School leaders: Colour-coding and confidence intervals 
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Most school leaders agreed that colour-coding of similar schools was useful (59% agreed; 14% 
strongly agreed), but there were some reservations about the way that the colour codes classified 
schools: 
“Colour coding of NAPLAN is necessary for accountability to the community. We use the data 
completely differently.” 


“| like the colours because at a quick glance | can see what’s happening. | don’t agree that it 
brings schools down.” 


“Don’t much like the colours but they focus attention.” 


Most school leaders (69%) also agreed that confidence intervals should be used on the Graphs and 
Student Gain pages. 


“NAPLAN is a point in time, so confidence intervals should be shown.” 


“There would be no loss in dropping off box plots because teachers and parents don’t 
understand them.” 


Unintended or negative consequences 


There has been substantial public comment over the years on the unintended or negative 
consequences of publishing school-level achievement data on the My School website, as well as 
other derivative presentations that appear on the internet and in newspaper league tables in some 
jurisdictions. It has also been argued that the pressure on schools to do well in NAPLAN may have 
led to narrowing the school curriculum, teaching to the test or putting students under unnecessary 
pressure. School leaders’ views on these issues are summarised in Figure 36. 
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Figure 36: School leaders: Impact of NAPLAN 
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Few school leaders in sample schools agreed that the curriculum had been narrowed in their school 
in an effort to boost NAPLAN results. Almost all either disagreed (38%) or strongly disagreed (55%) 
that this had occurred. None of the sample school leaders agreed that there had been narrowing of 
the curriculum in their school, but some had heard of it happening in other schools: 
“Definitely has not led to narrowing the school curriculum. If you teach well, you will do well 
in NAPLAN.” 


“We have broadened the school focus on the arts, and there is a whole-school focus on 
literacy. We have common processes, a literacy block, common language. The focus is on 
pedagogy.” 

“No way. Not a big focus on the test. Focus is on teaching and learning then over time kids will 


do well on things like NAPLAN.” 


“Staff didn’t make the connection between NAPLAN and he Australian Curriculum. We’re not 
teaching to the test, we’re teaching the curriculum in a focused and methodical way. So, we’re 
seeing improvement in both NAPLAN and PAT.” 


School leaders did agree, however, that their schools had focused within the curriculum on areas of 
weakness: 
“NAPLAN allowed us to see weaknesses, but now we look more deeply at each cohort using 
PAT.” 


“No, we have focussed on writing. We’ve looked at the data, focused, and built the capacity of 
teachers.” 


“Some reshuffling of teaching content to ensure that students have been taught all the text 
types.” 


“Some classroom writing tasks that echo NAPLAN writing tasks, but these are all good bread 
and butter things to teach.” 


“Higher rotation of mathematics strands, which is a good thing.” 


“Has not led to the narrowing of the curriculum here, but | know of schools where it has.” 
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Few school leaders in sample schools agreed that students had spent too much time preparing for 
NAPLAN tests. Almost all either disagreed (45%) or strongly disagreed (38%) that this had occurred. 
School leaders reported that there was little NAPLAN test preparation beyond familiarising students 
with test formats and timed tests. Some had heard of higher levels of test preparation in other 
schools: 
“We do some practice, e.g. in use of timed tests which they will also encounter at secondary 
school and university.” 


“We spend more time on how to read questions, test skills in Year 3. It’s a few hours over all.” 


“Preparing them for the future as learners requires teaching them how to answer multiple 
choice questions.” 


“Not in this, however yes at other schools | have worked at.” 


About half of school leaders agreed or strongly agreed that students had been too stressed about 
the NAPLAN tests. Many disagreed (41%) or strongly disagreed (10%) that students had been too 
stressed. Where they had encountered stress, they were inclined to attribute to students’ personal 
characteristics or parental pressure, rather than the school’s practices: 

“There may be some pressure from home to do well. This is not pressure about NAPLAN 


|.” 


specifically, but general pressure to do wel 
“Kids who need support are those who normally need support.” 

“Some kids are nervous little souls.” 

“We try to make sure kids are not stressed.” 

“One parent was concerned last year, but with an easily stressed child.” 


“There is some stress for kids, but many are very excited to do NAPLAN.” 


A further area of concern has been about potential misuse or misunderstanding of school-level 
NAPLAN data when families are choosing schools for their children. School leaders were asked 
whether they had any experience of this, either through use of My Schoo! NAPLAN data or 
secondary interpretations in other internet rankings or newspaper league tables. Their responses 
are summarised in Figure 37. 
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Figure 37: School leaders: NAPLAN and school choice 
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School leaders in sample schools had mixed views about the impact of NAPLAN reporting on school 
choice. A few school leaders (10%) thought that newspaper league tables had had an impact on 
families choosing or leaving their school (although 31% said that they did not know). 
“The use of My School to create league tables in the media is disappointing, as | feel it can be 
negative for schools that are seen to be low performing.” 


A few school leaders agreed that other internet rankings had had an impact on school choice (21% 
agreed). The remainder disagreed (17%), strongly disagreed (28%) or did not know (34%). 


Among these sources of school-level NAPLAN data, the highest level of concern was expressed about 
the My School NAPLAN displays. More than a quarter (28%) agreed or strongly agreed that families 
had chosen or left the school based on My Schoo! NAPLAN results. The remainder disagreed (24%), 
strongly disagreed (24%) or did not know (24%). 
“| don’t like the NAPLAN results on My School. Lots of families from outside the boundaries 
want to get in and we have real estate agents calling to find out which side of the street is 
within the school boundary. We want kids to do as well as they can, but don’t like the 
negative feedback because of our high performance. We get negative comments about 
parents paying for coaching. One principal said to me that it’s just because lots of your 
students are Chinese.” 


“Not aware of this. My School is never mentioned by parents.” 


“Movement away from the school tends to be about bullying, children’s unhappiness. 
Learning has never come up in these conversations.” 


“A couple of interstate parents have looked at My School and mentioned it in emails when 
they have asked for a school tour.” 


“Some of our teachers have moved houses because of school catchment, and they look for 
other things such as music and sport that you can’t see from My School.” 
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“Have had one or two parents mention NAPLAN when their child was starting Prep, and one a 
couple of years ago who left because of NAPLAN.” 


“This term we have had five or six new enrolments, half of whose parents had looked at 
NAPLAN data and mentioned it. One parent from [a nearby town] was shopping around and 
moved his child here, mentioning NAPLAN. None of those who left mentioned NAPLAN in exit 
interviews.” 


How NAPLAN Results are communicated 


School leaders were also asked to comment on how they communicated school-level NAPLAN 
results to students, parents and the wider community. Some restricted their communication of 
NAPLAN results to reports to school councils or information sessions: 

“We report NAPLAN results to parents in school newsletters but we don’t make a big fuss.” 


“Board Report includes NAPLAN and Scout analyses.” 


“Communicate to the Board and school community through the operational plan, which 
includes reference to NAPLAN.” 


We report on NAPLAN to the P&C, both good and bad results. They talk about gains and areas 
for improvement.” 


“Have held information sessions for parents but only five came the year before last and three 
last year, so | have stopped doing it.” 


A few schools, including a public and an independent school aiming to increase student numbers 
were more forward in their communication: 
“Do show off a bit to staff and the school board when the results are good, but we don’t put it 
on Facebook or in the newsletter.” 


“We put the ACARA letter everywhere! School website, Facebook, front office.” 


“We got an ACACA high gain letter the last two years. We put this information on Facebook 
and in parent newsletters. This led to some conversations with parents about how well the 
school has done and the hard work of the teachers.” 


TEACHERS 


Classroom teachers were asked to comment on the same issues as their school leaders: purposes 
served by the current My Schoo! NAPLAN data presentations, the use of My School and other 
alternatives for improving learning, the value of particular My Schoo! presentations, the use of 
colour-coding and confidence intervals on the graphs and potential for negative uses of NAPLAN 
data. Teachers’ views about the usefulness of My School and other data presentations were broadly 
consistent with those of school leaders in the sample schools, but teachers were generally a little 
more likely to be cautious or concerned. 


Page 49 


Purposes for using My School NAPLAN data 


Most teachers agreed or strongly agreed that the My School data presentations were useful for 
school improvement (81% agreed or strongly agreed), Identifying trends and gaps in achievement 
(76% agreed or strongly agreed), making comparisons between schools (71% agreed or strongly 
agreed), accountability for student achievement (67% agreed or strongly agreed) and Informing 
teaching practice (63% agreed or strongly agreed) (see Figure 38). 

“NAPLAN has a place — to be accountable.” 


“Yes, NAPLAN is used for target setting, particularly the number of children in the upper two 
bands and number moving between bands.” 


“It assists us to work out our strategic plan and provides teachers with reasonable data to 
improve individual students’ capacity or provide professional development for teachers.” 


“School planning reflects NAPLAN (top two bands, percentages, milestones) We spend more 
time on how to read questions, test skills in Year 3. It’s a few hours over all.” 


“Have always looked at it when looking at a new school appointment.” 


Teachers were less likely to agree that the My School information was useful in setting targets for 
students (57% agreed or strongly agreed) or informing school choice (45% agreed or strongly 
agreed). 
“There needs to be standardised testing as the government needs to be held accountable and 
have a way of measuring student performance. The NAPLAN data does not need to be 
available to parents on My School as it promotes competition and anxiety in our students.” 
“NAPLAN data should be made available to schools, not put on the web. Schools should not be 


[2 


able to compete with one another — a comment based on what | saw in another schoo 


“School comparisons are insulting. Every teacher does a service and different levels of 
achievement doesn’t indicate the success of a school.” 


“The graphs that show change are far more useful for a teacher. We know where our students 
are and teach accordingly. | feel some parents could use comparison data to make unfair 
judgements about schools.” 


“My School shouldn’t compare schools; it should be internal to the school.” 
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Figure 38: Teachers: Purposes for My Schoo! NAPLAN data presentations 
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Standardised assessment dashboards 


Teachers reported that they found a range of data presentations useful in understanding student 
progress and achievement (see Figure 39). Most found the My Schoo! data displays useful (My 
School: 80% agree or strongly agree); school system: (86% agree or strongly agree). Other 
standardised data displays such as those provided by the ACER PAT tests were almost universally 
regarded as useful (43% agreed; 53% strongly agreed). 
“Keen that leaders understand the data well, but I’m not likely to get that from looking at My 
School.” 


“We look at Scout as a staff, but | have only looked at My School as a parent.” 


“Have looked at sub-strands where we were above and below and changed teaching as a 
result.” 


“There has been broad school improvement in spelling after seeing the NAPLAN results.” 


“We use Scout in staff meetings. We have good conversations about which kids have 
progressed and which have not.” 


“| don’t feel NAPLAN data on My School is helpful to me as a teacher. However, NAPLAN data 
in OneSchool is used extensively.” 


“We look at raw information from NAPLAN item by item. It’s good to know what sort of 
things they need to know in NAPLAN for future years.” 


“We use NAPLAN for identifying trends and gaps and priorities. Use PM benchmarks for target 
setting for students.” 


“We use PAT for item level analysis, grouping, and planning curriculum emphasis.” 
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Figure 39: Teachers: Understanding student progress and achievement? 
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Specific My Schoo! NAPLAN pages 


Most teachers regarded all of the My Schoo! data pages as useful (Figure 41). The Graphs page was 
the highest rated; 84% of teachers agreed or strongly agreed that it was useful in understanding 
student progress and achievement. Student Gain (82% agreed or strongly agreed), Numbers (75% 
agreed or strongly agreed) and Bands (75% agreed or strongly agreed) were also widely regarded as 
useful. The least well-regarded data display was the Similar Schools page (49% agreed and a further 
14% strongly agreed). The following comments reflect the range of teachers’ views: 

“Data should represent growth or display areas that need support for a particular school and 

not be used to compare schools with each other.” 


“Shouldn’t be making school choice decisions solely on NAPLAN.” 

“| like the accountability and school improvement aspects of My School.” 

“Everyone’s entitled to this information, so that they can compare apples with apples.” 
“So much money spent on schools you might as well make it accountable.” 


“| have looked at NAPLAN as a parent, as a staff member and as a job seeker writing to go 
where good things are happening.” 
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Figure 40: Teachers: My School data presentations 


Teachers: Which My School data presentations are useful in understanding 
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Colour-coding and representation of measurement uncertainty 


As was the case with their school leaders, classroom teachers were generally supportive of colour- 
coding to show relative performance of school, with 78% agreeing or strongly agreeing that this was 
useful (Figure 41). As one teacher commented: 
“Parents in our community would find the colour coding easy to understand — they can access 
whole school info without having to analyse it.” 


“Colour-coding highlights the data, helps people to read the graphs. For example, it draws 
attention to our Year 5 spelling.” 


“On My School, colour coding makes the graphs easier to read. They are helpful because it 
tells us where we need to improve.” 


Teachers were more cautious about the use confidence intervals in the representation of 
measurement uncertainty. Although 55% agreed or strongly agreed, many either disagreed or did 
not know. Some reported that they “take no notice” of confidence intervals and “would be happy to 
see them deleted.” 


Figure 41: Teachers: Colour-coding and confidence intervals 
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: Unintended or negative consequences 


Teachers’ views about unintended or negative consequences of publishing school-level comparative 
NAPLAN data were broadly consistent with those of their school leaders (see Figure 42). Most 
teachers disagreed that the school had narrowed the curriculum to boost NAPLAN results (disagreed 
44% and strongly disagreed 36%). More often, they argued that NAPLAN results help them focus, 
rather than narrow the curriculum, because “we still need to teach everything.” 


Figure 42: Teachers: Impact of NAPLAN 
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Although teachers generally rejected narrowing the curriculum in their own school, there were some 
who disagreed. As one put it: 
“NAPLAN drives teaching in first term. There’s pressure to get results. It’s driven by admin 
staff, not negatively but it is a priority: literacy in the morning, numeracy in the afternoon.” 


More often, teachers commented that narrowing the curriculum was occurring in other schools. As 
several teachers put it: 


“Not at all in this school but have heard it said about other schools.” 


“When | was teaching in (a school in another state) every lesson and every assessment had to 
reflect NAPLAN.” 


“Have been in a school that was really centred on NAPLAN. Once it went on My Schoo! there 
was pressure for principals to perform, continual meetings on NAPLAN, and it led to NAPLAN- 
focussed teaching.” 


“We don’t teach to the test. | have friends in other schools that were practicing for the 
NAPLAN writing test.” 


Almost all teachers disagreed (49%) or strongly disagreed (31%) that too much time had been spent 
preparing for NAPLAN tests. Teachers made comments such as: 


“For year 3 it is their first timed test, so they need a bit of practice at that.” 


“Students have spent too much time? Not in this school, however ‘yes’ at other schools | have 
worked at.” 
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Teachers were evenly split between agreeing or disagreeing that students had been too stressed by 
NAPLAN. Among the 49% who agreed, 22% strongly agreed; among the 47% who disagreed, only a 
few strongly disagreed (6%). Teachers who were not concerned about too much stress mentioned 
that students “need exposure to test conditions”, argued that “there’s no pressure on teachers or 
kids here” or looked beyond the school suggesting that “There’s a beat up on social media about 
student stress.” As teachers put it: 

“We just tell them to try their best.” 


“We tell the kids that NAPLAN is just another day. Not the be-end-all.” 


“We've had a few kids that were in tears the day before NAPLAN. It didn’t come from us. We 
don’t make a fuss about NAPLAN.” 


“Yes, several children have been quite stressed. Parents make a lot of it. Usually it is particular 
parents who think children will be stressed, or the anxious disposition of the child. The 
teachers’ role is to downplay it with parents.” 


“My kids looked forward to NAPLAN.” 


Teachers had little evidence of an impact of NAPLAN on school choice (Figure 43), but there was 
some variation among schools. A majority (51%) disagreed that My Schoo! NAPLAN data had an 
impact on families choosing or leaving the school, strongly disagreed (a further 20%) or did not know 
(18%). Almost all of those among the few who agreed or strongly agreed came from a single school. 
Responses were broadly similar for the impact on school choice in their school of other internet 
rankings (67% disagree or strongly disagree) or newspaper league tables (63% disagree or strongly 
disagree). 

“In terms of shaping parents’ views of the school social media beats this stuff hands down.” 


“No experience of this school or in the community more generally of schools being chosen on 
the basis on NAPLAN results.” 


“Have never heard anyone talking about NAPLAN in the context of school choice.” 


“Have heard in the media about parents using NAPLAN to choose schools but have never seen 
it. | have heard parents talk about (Year 12) results. A couple have said that they have looked 
at a couple of schools when choosing high schools.” 
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Figure 43. Teachers: NAPLAN and school choice 
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How NAPLAN results are communicated 


Teachers reported that they sometimes discussed individual results with parents or talked students 
through their personal results before NAPLAN reports went home. 
“We talk individual kids through their NAPLAN report before it goes home to parents.” 


“We dedicate some parent / teacher interview time to NAPLAN.” 


“We look at NAPLAN results as a staff before individual results go home. We get in touch with 
a few parents who might be concerned by their child’s results.” 


“Teachers wrote a note to go home with children in each class, emphasising growth not just 
scores.” 


Teachers had little awareness of their schools communicating whole-school NAPLAN results to 
parents beyond school newsletters and reports to school boards. 
“Don’t make a big deal of it.” 


“NAPLAN results are reported to the school board.” 
“NAPLAN results might be mentioned in the school newsletter.” 


“NAPLAN info goes in the newsletter and a report goes to school council.” 
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PARENTS 


Parents attending the focus group interviews were asked to comment on the potential uses of 
NAPLAN data, their experience of any negative consequences of My School or other NAPLAN data 
displays and how their school communicated to them about NAPLAN results. Fewer than a half of 
parents had ever looked at My Schoo! before they attended the focus group (Figure 44). When their 
school’s My School NAPLAN data pages were made available at the focus groups, however, they 
were interested to read and analyse on the data displays. 


Figure 44: Proportion of parents with prior experience of My School 


Parents: Had looked at My School Before Today 


= Had looked at My School 


= Had not looked at My School 


Purposes for using My School NAPLAN data 


Most parents were supportive of school-level NAPLAN results being available on a public website, 
with a total of 81% who agreed or strongly agreed (Figure 45). As one parent put it: “How else would 
you know if there were any problems in the school?” Parents were also likely to agree that the My 
School NAPLAN data presentations were useful for the purpose of choosing schools (68% agreed or 
strongly agreed) and reassuring them that the school is doing a good job (70% agreed or strongly 
agreed). 


In most school focus groups, parents reported that school NAPLAN results rarely were mentioned in 
the family and community conversations. In one notable exception to this general trend, parents at 
a very high performing school acknowledged that “People have moved here because of the 
rankings” and that this had meant the arrival of “a new group of people at the school.” In this 
school, focus group participants volunteered that “Chinese and Korean parents talk about NAPLAN 


results.” 
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Figure 45: Parents: NAPLAN and My School 
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Although a majority of parents regarded the use of NAPLAN data for school choice as legitimate, 
many were disinclined to agree that NAPLAN data had a substantial role in their school choice. 
“The information is interesting and useful but didn’t play a large part in determining to choose 
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this school for our children. It was the local schoo 


“| have three children and have never looked up results for either our school or any other 
school in the years my children have done NAPLAN. NAPLAN results have never persuaded me 
in my choice of where | send my kids to school.” 


“| looked up My School each time a child started school. It didn’t make me change my mind 
but it did reassure me.” 


A few parents mentioned looking at My Schoo! when the family was relocating to a new area: 
“When we moved into the area, we looked at rentals based on schools’ NAPLAN, their student 
numbers and the size of schools. We used My School a lot to identify good areas and then 
came to the schools for a visit. We used it for a first pass.” 


“We’re moving to Darwin, so | looked up My School, looked at the school population, the size 
and their NAPLAN results.” 


Parents who disagreed with the use of NAPLAN in choosing schools preferred to see NAPLAN as an 

improvement tool. 
“| use my child’s NAPLAN results as a tool to track the learning growth. i.e. moving up the 
bands. If there are areas of concern, | approach teaching staff to see if there is a problem or 
not. |am aware that it is a point in time test and a child could be having a bad day. | see it as 
an effective tool for schools to track data across the years. They can use this data to establish 
the progress of the intervention programs put in place in earlier years to support the 
downfalls that can be seen i.e. a drop in numeracy. | am interested in how schools use the 
data to help students gain. | have no interest in comparing NAPLAN results from different 
schools as a way to choose a school because each cohort of children is different.” 
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“For me the interest in NAPLAN is the relative performance of my child, her strengths and 
weaknesses and over time her improvement. The population element of this data should be 
used to track the child, not to give parents the choice between schools.” 


“Don’t approve of choosing schools on NAPLAN because some people don’t have a choice — 
people in rural areas and low SES areas. It would be more positive if parents in schools with 


[” 


low NAPLAN got involved and helped the schoo 


About two-thirds (68%) of the parents attending the focus groups were aware of the school 
providing feedback about school-level NAPLAN results, especially those parents who were members 
of their school boards or councils. 
“There is a leadership report to the school council. It was interesting and showed that the 
teachers hard work was paying off, it showed how reading has improved and now numeracy is 
being tackled.” 


“| believe NAPLAN is a very positive tool, but only if used effectively. At our school we have 
excellent staff who use the results to tailor specific teaching programs. NAPLAN is great when 
the results are interpreted well. Our school is great at communicating with parents.” 


“The school tells us about how they have gone in NAPLAN. They reported progress in literacy 
and mathematics. (School leader in charge of data) presents the results to the P&C in graphs. 
He uses it to show how kids improve. We can see where individual students are struggling. 
You’ve got to have someone who is good at interpretation. People bag NAPLAN but as long as 
it is interpreted, it is good. We get that in the school council.” 


| Specific My School NAPLAN pages 


Fewer than a half of the parents attending the focus group interviews had previously looked at the 
My School website, but after they had had a close look in the context of the focus group interviews, 
they almost always agreed that the My Schoo! NAPLAN pages were useful or very useful (Figure 46). 


Figure 46: Parents: My School data presentations 


Parents: Which My School data presentations are useful in understanding 
student progress and achievement? 
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Unintended or negative consequences 


There was little evidence that parents were aware of unintended or negative consequences of 
NAPLAN testing such as narrowing the curriculum, focussing on test preparation or unnecessarily 
stressing children (Figure 47). 


Figure 47: Parents: Impact of NAPLAN 


Parents: Impact of NAPLAN in this school 


This school has spent too much time preparing students kx Xx 0Ulmlti“‘i~S 
for NAPLAN 
Children at this school have been too stressed about i se 
NAPLAN 
This school has narowed the curriculum to boost _——~— 7 «| 
NAPLAN results 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Strongly disagree m Disagree Agree i Strongly agree Don't know 


Very few parents reported that their children had been too stressed by NAPLAN (46% disagreed; 
38% strongly disagreed). 
“NAPLAN can be stressful, but the tests help parents to see what can be done with their 
children’s learning.” 


“Have heard that children get stressed but have not experienced it.” 


“(A parent) was a bit stressed about opening the NAPLAN result for her daughter for the first 
time, in Year 3.” 


Nor had parents noticed any evidence that the school had narrowed the curriculum (41% disagreed; 
35% strongly disagreed) or spent too much time preparing for NAPLAN tests (65% disagreed; 22% 
strongly disagreed) 
“Have heard that some schools spend too much time preparing for NAPLAN but it definitely 
doesn’t happen here.” 


“School should already be teaching these things; they are core learning areas.” 


“Not a lot of practicing here, but there is in the school one of my children goes to.” 


STUDENTS 


Students attending the focus groups interviews were asked to comment on how important they 
thought NAPLAN tests were, who they were important to, what had been their experience of 
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teachers’ preparation for the tests they had sat, and what had been communicated to them about 
NAPLAN results. 


Importance of NAPLAN 


Almost all students thought that was important to do well in NAPLAN (Figure 48) and almost all 
reported that they NAPLAN was important to their teachers (46% important; 52% very important): 
“Important to teachers because they want to know the level of skill we are at.” 


“Teacher wanted us to do really well because it’s the last year before high school.” 


“Important because she (our teacher) wants us to do our best.” 


Figure 48: Students: How important is it to do well in NAPLAN tests? 
Students: How important is it to do well in NAPLAN tests? 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
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A similar high proportion thought that doing well in NAPLAN was important to their families (52% 

important; 33% very important), although there were some dissenting comments in the interviews: 
“My family wanted me to get a really good score because I’m going to (a private school) next 
year. They were really interested.” 


“Very important. My parents are very strict.” 
“My mum doesn’t believe in it.” 


Almost all students as important (62%) or very important (32%) to themselves that they do well in 
NAPLAN. As one student put it: “It’s important for me because | like to do my best.” Similarly, many 
thought it was important (53%) or very important (35%) to the school that they did well in NAPLAN 
because “It might teach the school what needs to be taught.” Some older students were also aware 
of the impact of NAPLAN results on the school’s reputation: 
“The school is rebranding, changing uniforms, marketing a lot more. More kids will come if we 
do well in NAPLAN.” “ 


It gives an idea where the school is at compared with others, but teachers play it down, don’t 
really talk about it.” 
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Students’ Experience of NAPLAN 


Students were also asked to comment on their experience of preparation for the NAPLAN tests, 
stress and teachers’ feedback on results (Figure 49). 


There was little evidence among those students selected by their schools for the focus groups of 

negative or unintended consequences. About one-third (37%) agreed that their class spent a lot of 

time practicing for NAPLAN. More often, they reported one or two lessons of test preparation: 
“We did a few questions from the last year’s test. It was about two pages of questions.” 


“We practised a few times. The teacher had a timer.” 
“One practice test in writing and one in maths. None in reading.” 


“We had one practice using the computer.” 


Figure 49: Students’ experience of NAPLAN 


Students: Experience of NAPLAN 


My teacher told the class how we had gone in NAPLAN 


My family was worried about how well | would do in 
NAPLAN 


| was worried about how well | would do in NAPLAN 
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About one-quarter of students (23%) acknowledged that their parents were worried about how well 

they would do in NAPLAN. As one student put it, families “are interested but not stressed.” Students 

were more likely to worry personally that they would do well (34% agreed). One-third of the 

students (34%) agreed that teachers had provided feedback on class or school NAPLAN results: 
“Really brief, like an acknowledgement, and only about writing.” 


“Teachers talked about it on the day and went over the questions afterwards.” 


Page 62 


SUMMARY 


UNDERSTANDING OF STUDENT PROGRESS AND ACHIEVEMENT 


Most teachers and school leaders in the sample schools agreed that the My Schoo! NAPLAN pages 
were useful in school improvement (85% of school leaders and 81% of teachers agreed that My 
School pages were useful for this purpose). They also agreed that the displays were useful for 
accountability (69% of school leaders and 67% of teachers), identifying trends and gaps in 
achievement (74% school leaders and 76% of teachers) and informing teaching practice (69% of 
school leaders and 64% of teachers). 


Their views differed on the usefulness of these displays for setting targets for students (69% of 
school leaders and 58% of teachers agreed) and making comparisons between schools (58% of 
school leaders and 71% of teachers agreed). Those teachers and school leaders who rejected the use 
of My School for comparisons argued that that there is more to a school’s success than NAPLAN or 
that comparisons may undermine schools. 


Neither school leaders nor teachers thought that the My Schoo! NAPLAN displays were useful for 
informing school choice (46% of school leaders and 44% of teachers). 


Fewer than half of parents (45%) involved in the focus groups had previously looked at the My 
Schoo! NAPLAN data displays. Most (84%), however, were supportive of school-level NAPLAN results 
being available on a public website. Most of the parents (68%) agreed that the data displays were 
useful in choosing schools and reassuring them that the school is doing a good job (74% agreed). 


USING ACHIEVEMENT DATA TO INFORM TEACHING 


School-level data displays of NAPLAN have developed substantially in the decade since My School 
was first published. Teachers and school leaders now have access to sophisticated school-sector and 
commercial test dashboards in addition to the My School data displays. 


In every one of the sample schools, teachers and school leaders displayed a deep understanding of 
their students’ achievement data and showed that they actively used it for school improvement 
purposes: identifying trends and gaps, setting targets for students and informing teaching practice. 


All school leaders and almost all teachers (96%) agreed that standardised test dashboards such as 
those provided for the ACER PAT tests are useful in understanding student progress and 
achievement. Slightly fewer people agreed that their school system dashboards are useful for this 
purpose (school leaders 88%; teachers 89%). Most school leaders (84%) and teachers (80%) agreed 
that the My Schoo! NAPLAN pages were useful for understanding student progress and achievement. 
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Asked to consider which of the My Schoo! NAPLAN pages are useful in understanding student 
progress and achievement, most school leaders (>85%) and teachers (>75%) agreed that the 
Numbers, Bands, Graphs and Student Gain pages were useful. Neither school leaders nor teachers 
thought that the Similar Schools page was as useful in understanding student progress and 
achievement (school leaders, 71% agreed; teachers, 62% agreed). Most parents (>85%) agreed that 
all of the My Schoo! NAPLAN data pages were useful for this purpose. 


REPORTING NAPLAN DATA TO STUDENTS AND PARENTS 


For all of the public commentary about NAPLAN, the results of the tests had a relatively low profile 
in the sample schools. A third of the students in focus groups (32%) recalled their teachers providing 
any feedback on class or school NAPLAN results. Those who could, recalled their teacher going over 
the questions after the test or acknowledging good performance when the results were ready to be 
sent home to parents. 


Teachers reported that they sometimes spoke to individual students about results, but they “don’t 
make a big deal of it.” Most parents (68%) recalled the school providing feedback through 
newsletters or school board reports. 


School leaders reported communicating NAPLAN results through school newsletters and reports to 
school councils. In a few cases, very good NAPLAN results or the ACARA high gain letter were posted 
on school Facebook sites. 


PERCEPTIONS OF MY SCHOOL AND NAPLAN REPORTING 


Although most parents reported that the My Schoo! NAPLAN displays could be useful for choosing 
schools, few of those consulted in the focus groups thought that My School was often used for this 
purpose. Other factors such as proximity, community feedback and school culture were much more 
important to focus group parents. 


Few school leaders had evidence that NAPLAN had an impact on school choice in their current 
school. About one fifth (19%) of school leaders were aware of parents having chosen or left their 
school on the basis on My School NAPLAN reporting or other internet rankings. None were aware 
that newspaper league tables had an impact on choice in their schools. 


Few teachers were aware that NAPLAN reporting had an impact on school choice in their school. 
None agreed that parents had chosen or left the school on the basis of My School data or internet 
rankings, but (4%) agreed that newspaper league tables had led parents to choose or leave their 
current school. 


The use of colour-coding to show whether a school’s NAPLAN results were higher or lower than 
expected is designed to make the graphs easier to interpret but was also thought to open up the 
possibility of naming and shaming lower achieving schools. Notwithstanding this, 77% of school 
leaders and 80% of teachers and agreed that colour-coding for achievement and growth should 
continue. As one school leader put it: “I don’t like the colours, but they focus attention.” 
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Among the potential negative or unintended consequences of NAPLAN, three were discussed in the 
focus groups: stress for children; teaching to the test and narrowing the school curriculum. Most 
school leaders (92%), teachers (77%) and parents (74%) disagreed that the curriculum had been 
narrowed, although they sometimes suggested that it was happening at schools other than their 
own. Most also disagreed that too much time was spent on test preparation (school leaders 81%; 
teachers 78%; parents 88%). Most parents (88%) were disinclined to agree that students were too 
stressed, but about half of the teachers (53%) and a third of school leaders (38%) agreed that 
students were too stressed by NAPLAN. 
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CHAPTER 3: SUBMISSIONS AND STAKEHOLDER CONSULTATION 
OVERVIEW 


The third and fourth phases of data collection for this Review were conducted in March and April 
2019. 


The opportunity to make written submissions to the Review was advertised in February 2019. An 
Issues Paper was provided to stimulate and structure submissions. Thirty-four written submissions 
were received. More than half of these were from school system or sector authorities; the 
remainder included submissions from research and advocacy groups, parents’ associations, unions, 
principals’ associations and individuals. 


In addition, the reviewer met with 23 national stakeholder groups in March and April 2019. These 
included school system and sector authorities, unions, and stakeholders representing the views of 
parents and school principals in government, independent and Catholic schools. A total of 73 
stakeholder representatives attended these meetings. 


The discussion of stakeholder views and submissions that follows is structured around the questions 
posed in the Issues Paper. 


PERCEPTIONS OF MY SCHOOL AND NAPLAN REPORTING 


THE RIGHT TO HIGH QUALITY INFORMATION AND THE POSSIBILITY OF MISUSE 


The balance between the right to know and possibility of misuse was the most vigorously contested 
issue in the Review. Stakeholder views are reported by organisational type in the discussion below. 


School sectors and systems 


Submissions from government education authorities typically did not contest the balance between 
the right to high quality information and the risks of misinterpretation or misuse in current 
representations of school-level NAPLAN data on the My Schoo! website. 


The Australian Government’s submission, for example, noted that “Education Ministers have 
committed to the principles of accountability and transparency”, that “My School! has been reviewed 
several times and overall, parents and the community support NAPLAN data being on My School”, 
and that the Australian Government Department of Education and Training “supports the on-going 
publication of My School, including the presentation of school level NAPLAN data” (p. 2). Similar 
views were reflected in many of the stakeholder meetings with state and territory government 
school systems. 
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Among government authorities, Queensland and the ACT were more cautious about the balance 
between transparency and the unintended negative consequences of the representation of NAPLAN 
data on My School. The Queensland Government had commissioned a major review of NAPLAN in 
the Queensland context in 2018.*8 On the basis of this review, the Queensland government 
concluded that: 


“Accountability is an important aspect of our education system and Queensland supports 
continuation of a national mechanism to ensure appropriate monitoring of progress at the 
system level. The Queensland Government acknowledges that NAPLAN has provided 
assistance in tracking literacy and numeracy outcomes over time. However, the negative 
effects identified by the review are such that new and better ways of monitoring progress 
are required. Though it will take time and national agreement, it is time for NAPLAN to 
evolve” (p. 4).*° 


The “negative effects” reported in the Queensland review included diverting teaching time to test 
preparation; narrowing the curriculum and teaching to the test; impacts on the wellbeing of 
students and teachers; and a focus on shifting students at the edge of one band to the next band to 
improve aggregate results. The Queensland submission affirmed a commitment to ensure that “any 
reporting on NAPLAN reflects its limited role and does not lead to negative consequences such as 
impacts on wellbeing” (p. 2). 


The second jurisdiction to express reservations about the balance between transparency and the risk 
of negative consequences was the ACT. Their submission argued that although “the My School 
website does provide some high-quality information, there remains an unacceptably high potential 
for misinterpretation and/or misuse” (p. 2). The particular risk identified in the ACT submission 
concerned school choice: 


‘The concept of choice operating [on My School] assumes that education is a service for 
which there is a free market. However, for a very significant proportion of the population, 
there is little or no choice available, as the cost of non-government education is such that 
this group of people is precluded from taking advantage of non-government education.’ (p. 
2). 


This competition, the ACT submission argued, is “against the public interest due to the negative 
impact on public schooling ... [and] the contribution that this competition makes to social and 
economic division in Australia” (p. 2). 


Among non-government school sectors and systems, there were high levels of support for 
publication of school-level NAPLAN data but consistent reservations about the use of NAPLAN data 
for school comparisons. 


48 https://ged.qld.gov.au/programs-initiatives/education/naplan-2018-review 
49 https://gqed.qld.gov.au/programsinitiatives/education/Documents/naplan-2018-qld-gov-response.pdf 
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Submissions were received from several authorities in the Catholic education sector. The Townsville 
Catholic Education Diocese reported that “the current version of the My Schoo! website provides an 
appropriate balance between the right to high quality information and the possibility of 
misinterpretation or misuse.” The submission argued that all statistical information and graphical 
representations are “susceptible to misinterpretation” but that the comparisons available in the 
current version of My School “are better contextualised than in previous versions of the website” (p. 
1). 


The Queensland Catholic Education Commission affirmed that “NAPLAN provides useful feedback 
about student, school and system performance” but “does not provide a complete picture of a 
school community or a child’s learning.” The submission identified a series of advantages of the 
publication of NAPLAN data on My School, including accountability of reporting on a nationally 
consistent basis, information about achievement over time, visibility of school progress and 
achievements and capacity for like-school comparisons and assistance in parental choice. Against 
these benefits, the submission drew attention to significant risks of misinterpretation, over-reliance 
on a narrow source of achievement information, publication of league tables, student stress and 
narrowing of the school curriculum. For these reasons, the QCEC submission called for “further 
refinements and communications regarding context, collection and purpose of NAPLAN data on the 
My School website” (p. 3). 


The National Catholic Education Commission’s submission “supports a national co-ordinated 
approach to measuring student performance which benefits schools and provides valid, reliable and 
contextualised feedback for parents and families”, but was “concerned that NAPLAN is undermined 
by the making of simplistic comparisons ... and the use of NAPLAN results to market schools” (p. 2). 
Of particular concern was the fairness of similar-school comparisons made using ICSEA. In the 
context of the stakeholder interviews one of the senior officials from the NCEC commented that 
“there is more to be done but what we have done should not be abandoned.” There was no in- 
principle objection to the release of school-level NAPLAN data, “so long as it is done fairly.” 


Similar concerns about the fairness of school comparisons were raised by the independent school 
sector. Independent Schools Queensland, for example, “recognises the importance of transparency 
and accountability” but “does not support school or system comparisons” (p.4). Independent 
Schools Tasmania reported that “schools, parents and the wider community should have access to 
many elements of the current NAPLAN data” but had reservations about “the current comparison 


m 


tools presented between ‘similar schools’” (p.4). The national independent school body ISCA 
reported that their sector “recognises the importance of transparency and accountability” (p. 2) but 
was concerned that “My School has become a tool for rating schools in the community and media” 


(p. 3).°° 


5° School league tables were published early in 2019 in The Courier Mail and Adelaide Advertiser, but state 
government websites rather than My School appear to be the source of the data. Tables published by the 
Canberra Times do appear to have drawn data directly from My School. 
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Parents’ and principals’ associations 


Among the parents’ associations, the Australian Parents Council was a strong supporter of the 
current approach to presentation of school-level NAPLAN data of the My School website. 
Recognising the possibility of misuse of the data in league tables, their submission noted that “APC 
supports the access to and publication of NAPLAN data provided that it is done under strict and 
agreed guidelines” (p. 2). Their submission argued that “it is better to manage the use of the data 
than to deprive parents of information about their child’s school” (p. 2). 


Catholic School Parents Australia’s submission acknowledged that “a key use of the My Schoo! data 
is a check on a child’s school for overall performance and a comparison with other schools” (p. 1). 
The submission from one of CSPA’s state-based affiliates was more equivocal about the question of 
balance. Catholic School Parents Victoria noted that NAPLAN “has provided some transparency” and 
“provided a catalyst for parents to have higher expectations of our teachers and of schools to do 
something about it” (p. 6). They note, however, that the use of the My Schoo! website to compare 
schools “has in many ways created a skewed view in how parents interpret the quality of a school 
and the progress of student learning” (p. 6). 


The Australian Council of State School Organisations did not provide a formal submission but in the 
context of stakeholder consultation characterised ACSSO as “broadly supportive” of NAPLAN and My 
School, referring the reviewer to their national presidents’ public statement in 2018: 


“ACSSO is a long-time advocate for fair, simple and transparent education policy, and whilst 
we recognise the need to always reflect on the purpose and methodology of NAPLAN and 
the curriculum, at some point we have to trust that such initiatives are delivered with the 
best of intentions by principled advocates seeking the very best, within their remit, for our 


nation’s children.”?* 


Conversations with representatives of ACSSO’s state-based affiliates revealed some diversity of 
opinion. Written briefing notes provided by Parents and Citizens Queensland argued that “No 
NAPLAN data should ever be used in My School website.” In contrast, the briefing notes provided by 
the ACT Council of Parents and Citizens Associations reported that: 


ACT parents and carers have repeatedly called for NAPLAN and My School data to remain 
available, providing transparency and public accountability for public expenditure. Parents 
and carers consider NAPLAN and My Schoo! essential measures of both school and 
education system performance for government and the broader community. 


There was a similar range of views among principals’ associations contributing to the review. The 
national peak body for government, Catholic and independent primary school principals, the 
Australian Primary Principals Association, was supportive of the current presentations of NAPLAN 
data on the My Schoo! website. Their submission commented that “APPA recognises the desire of 


51 https://acaraweb.blob.core.windows.net/acaraweb/docs/default-source/assessment-and-reporting- 


ublications/acsso-president-39-s-message-feb-2018.pdf?sfvrsn=2 
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governments to be transparent with data and accountable to the community” and that “the My 
Schoo! website provides a solid overview of NAPLAN data” commenting that “the colour of cells and 
graph diamonds, for example, provides a good indication of school progress” (p.2). Notwithstanding 
this broad endorsement, they commented that “My School is susceptible to misuse”, drew attention 
to the role of jurisdictions in preventing data misuse and misinterpretation and questioned he value 
of the website’s “purpose or relevance in the modern school” (p. 2). 


The Association of Heads of Independent Schools Australia based their submission on a substantial 
survey of members. They reported that a majority (55%) of their principals do not agree that My 
School represents an appropriate balance between the right to high quality information and the risk 
misinterpretation or misuse and that 68% of Heads “are of the view that schools’ NAPLAN data 
should not be published on the My Schoo! website.” (p. 3). The key issue for AHISA principals was 
“misrepresentation of schools’ quality of provision” through issues including narrowness of what 
makes for a successful education, narrowness of scope in NAPLAN data, short-term views of 
progress, lack of indication of Year levels where there are substantial intakes and lack of context for 
the school’s philosophy (p. 8). 


In the context of stakeholder consultations, the Catholic Secondary Principals’ Association reported 
that NAPLAN was a valuable tool “for accountability within sectors, across sectors and for 
governments.” Their concern was with access to the data, because “as soon as it becomes public it is 
used to rank schools”. The Australian Secondary Principals’ Association argued that “accountability 
should be at the local level, with principals having autonomy about what data to present.” The 
Principals’ Federation of Western Australia’s submission contested whether a national assessment 
scheme is needed at all, given “the raft of information about student progress and achievement:” 
provided by schools (p. 1). Comments provided by the Australian Catholic Primary Principals 
Association varied across jurisdictions. Several state and territory representatives supported 
publication of NAPLAN data, commenting that “it is part of the culture we accept in schooling 
today.” Others took an opposing view, arguing that “the data does not need to be published.” 


Unions, schools and individuals 


Both of the national teachers’ unions made written submissions to the review. The Australian 
Education Union submission supported publicly availability of information about school and student 
performance “to track progress at a systemic level”, but argued that the construction and mode of 
delivery of NAPLAN tests “has repeatedly been shown to be unreliable and is presented through My 
School in such a way that it is frequently misinterpreted by parents” (p. 5). The manner of 
presentation of NAPLAN data on the My Schoo! website, the AEU argued, the makes NAPLAN “high 
stakes” for schools, encourages parents to judge schools on “a narrow range of measures” and leads 
to “distortions of teaching and learning processes.” They argued that “there is no good outcome 
from the publication of school-level NAPLAN data (p. 4). 


The Independent Education Union submission offered “in principle” support for the published 
purposes of NAPLAN data represented on the ACARA website but argues for mitigation or removal 
of a series of “negative impacts on curriculum, pedagogy, learning, student well-being and school 
enrolment behaviours” (pp. 3-4). The submission went on to say that there is “an irreconcilable 
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tension” between data that is useful for accountability and data that is useful to teachers (p. 5). The 
submission cited a 2019 IEU survey on NAPLAN where an “overwhelming number” of members had 
major concerns about NAPLAN and showed “little support” for the My Schoo! website. Members 
disagreed that NAPLAN provided an accurate evaluation of students’ abilities or was useful for 
planning student learning (p. 10). 


Research and advocacy groups 


Five research and advocacy groups provided submissions to the Review: the Gonski Institute for 
Education at the University of New South Wales, the Grattan Institute, the Centre for Independent 
Studies and the Institute for Learning Sciences and Teacher Education at the Australian Catholic 
University and The Smith Family. 


Three of these submissions supported the continued publication of NAPLAN data on My School, 
Grattan, The Smith Family and the CIS. The Grattan Institute argued that NAPLAN is “essential”, “a 
national asset” and “a vital tool”: 


‘Governments and other system leaders rely on standardised testing to understand which 
schools are struggling or thriving, and what interventions work well and should be 
expanded” (p. 1). 


They went on to argue that NAPLAN is useful for monitoring at the system level and supporting 
school leaders to make decisions about where to allocate their time and resources but is not the 
best tool for targeted teaching or stimulating improvement through competition between schools. 
Schools and teachers need detailed diagnostic information in order to identify what students need 
to learn next, or what difficulties they face. Such detailed diagnostic information, they said, goes 
beyond the capacity of NAPLAN to deliver, even with improved online adaptive testing. The best way 
to get the balance right between NAPLAN and diagnostic assessment “is not to throw out NAPLAN” 
but to strengthen “small data in the hands of teachers” (p. 8). Secondly, they argued, the capacity of 
My School to support parents’ school choice would be increased if the quality of the data on the 
website were improved. Options for improvement included a broader range of quality assured 
judgements about teaching quality; extending NAPLAN to include students’ general capabilities; and 
more focus in the NAPLAN displays on students’ gain compared with students with the same starting 
scores (p. 13). 


The Smith Family submission argued that access to quality data is the foundation of improving 
students’ educational outcomes and that NAPLAN and the My Schoo! website have been “an 
important part of that foundation and to improving the performance of Australian schools and 
students” (p. 3). The submission suggested, however, that more can be done in several ways: by 
aligning NAPLAN with learning progressions and student growth, by improving transparency of test 
design, the quality of communication to parents and rethinking the approach to minimum standards. 


The Centre for Independent Studies submission described NAPLAN as “a high-quality school, system, 


state and national overview of student achievement in foundational educational skills” (p. 3). The 
submission stressed that use of My School is voluntary, the website “contains numerous caveats” 
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and presents the data in in multiple formats, which “reduces the possibility of data misuse or 
misinterpretation” (p. 4). 


The Gonski Institute’s submission provided a detailed set of arguments for moving NAPLAN from 
census testing to sample testing. An appendix to the submission argued that publication of NAPLAN 
data on My School had led to a variety of unanticipated outcomes including use of NAPLAN as a basis 
for admission to independent and selective schools; development of test-preparation products and 
coaching schools; generation of league tables; and impact on real estate values. In addition, it was 
argued that the use of NAPLAN data on My School! had led to narrowing of the curriculum and 
teaching to the test and had an impact on students’ and teachers’ wellbeing. 


The Institute for Learning Sciences and Teacher Education’s submission provided a comprehensive 
critique of NAPLAN and My School. It argued that NAPLAN reporting on My School does not provide 
an appropriate balance between high quality information and the possibility of misinterpretation of 
misuse. It contested the quality of the tests as “out of step with international tests of literacy and 
numeracy” (p. 4); argued that My Schoo! has made NAPLAN “high stakes”; expressed concern about 
statistical or technical issues in interpretation of My School data pages; and cited the publication of 
league tables as misuse of the data. 


STATISTICALLY SIMILAR SCHOOL COMPARISONS 


A range of views were expressed about the quality of the explanatory material on similar schools. 
The CIS said that the explanatory material is “both easy to understand and sound from a statistical 
perspective” (p. 6). Notes provided by ACSSO included the ACT Council of Parents & Citizens 
Association’s comment that “Council considers that ‘statistically similar schools’ is well understood 
and allows for reasonable comparison between like schools”. The ISQ submission noted that 
although there are questions about the validity of using ICSEA for comparisons, “the availability of 
explanatory material on the My School website is adequate for those seeking further clarification” 
(p. 5). 


The Australian Parents Council, on the other hand, said that statistically similar schools were “not 
well explained for parents” and the information that is available on My School “is difficult to find” (p. 
2). Catholic School Parents Australia’s submission noted that there is “no general understanding of 
what a ‘like school’ is for parents” (p. 2). The ISCA submission acknowledged that while My Schoo! 
includes explanatory materials regarding similar schools, “it is debateable whether users can locate 
these or whether they find the materials useful” (p. 4). 


Many of the Catholic and independent school sector submissions contested the appropriateness of 
making school comparisons, but their concerns were not with the quality of the explanations used 
on My School! but the statistical basis of the comparisons or the usefulness of comparisons of any 
kind. ISCA, for example, contested the appropriateness of statistically similar school groupings, 
noting difference in context, size, demography and geographical location among schools with similar 
ICSEA values (p. 4). 
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Principals’ associations all expressed concerns about the appropriateness of school comparisons. 
Almost all (90%) of principals responding to the AHISA survey reported that the ICSEA-based 
determination of statistically similar schools does not support fair comparisons for schools (p. 12). 
Respondents to the AHISA survey took issue with technical issues underlying ICSEA, including self- 
reported data and the categorisation of parents’ occupational levels. In addition, they argued that 
ICSEA does not capture the characteristics parent wish to compare, such as cultural context, 
geographical location or whether the schools are single-sex or boarding schools. 


The Catholic Education Commission of Victoria’s submission focused of technical issues around 
statistically similar school comparisons. Their primary concern was that the colour-coding of results 
on My School was based on ICSEA scores and that the ICSEA methodology failed to take account 
differences in the main language spoken at home. A detailed technical paper provided in the 
submission showed that students from some language and cultural backgrounds performed better 
than expected and some performed worse than expected (pp. 4-7). Without adjusting for these 
differences, CECV argued, ICSEA based similar school comparisons would consistently under- or 
over-estimate school performance (pp. 11-18). Comparisons based on student gain in schools “are 
also distorted by student cultural/language background” but the effect size of student 
cultural/language background on student gain (after adjusting for starting score) is usually much 
smaller than its effect on student mean NAPLAN scores.” For this reason, CECV recommended that 
My School places “much greater emphasis on student gain data than school mean NAPLAN scores 
and school ICSEA values” (p. 19). 


Somewhat similar concerns were raised in the ACT Government submission, which drew attention to 
several ways in which ICSEA-based similar school comparisons may not be fair. At a jurisdiction-wide 
level, the ACT argued that there a systematic “upward bias” in ICSEA scores as a result of the over 
representation of degree qualifications, or higher, among families in the ACT. Secondly, the ACT 
submission expressed concern about inclusion of academically selective schools in ICSEA-based 
comparisons because “performances of schools with a similar ICSEA to a selective school can have 
results that vary very substantially” (p. 3). 


ILSTE raised two further concerns about calculation of ICSEA, that parent education and 
employment were self-reported and the absence of variables that contributed to educational 
outcomes: student gender, language background and proportion of students with a disability in a 
school (p. 7). QCEC also argued that that explanatory material around statistically similar schools 
could be improved, especially with regard to student disability and language background (p. 4). 


In the context of its broad objections to school comparisons based on NAPLAN, the AEU argued that 
the omission of financial information from comparisons “has the effect of obscuring the huge impact 
that funding and finance have on performance” (p. 10). 


INFORMATION INCLUDED ON MY SCHOOL 


This question from the Issues Paper drew two kinds of responses, comments about the language and 
graphics currently on My Schoo! and comments about the amount of information that should be 
provided. Views on the suitability of language and graphics varied. The CIS submission rated the 


Page 73 


language on My School as “easy to understand” and benefited from the use of colour coding to aid 
interpretation (p. 5). Similarly, the QCEC submission reported that NAPLAN data reported on My 
School are “generally considered to be user-friendly and clearly formatted with graphics that are 
informative and well designed” (p. 3). 


In contrast, ISCA noted that although the information on My School is hyperlinked to fact sheets, 
“the language is not user-friendly and presumes prior knowledge, for example, of statistical 
comparison” (p. 4). Principals responding to the AHISA survey reported that parents had most 
difficulty understanding the Similar Schools display and Student Gain pages and least difficulty 
understanding the Bands and Numbers displays on My School (p. 9). Other submissions identified 
the range of potential audiences as a problem: 


“Meeting the needs of a wide range of audiences has meant that the site lacks a clear 
pathway for users to access information. Some of the detailed and technical information is, 
for example, difficult to locate, with interpretation requiring a high level of data literacy” 
(AISSA, p. 2). 


In terms of content, many stakeholders commented that there was too much emphasis on NAPLAN 
on the My Schoo! website. One option for change was to reduce the number of NAPLAN displays 
available. Most commonly, school systems and sectors would be satisfied if My School displays were 
reduced to achievement and student progress (currently represented in the Numbers and Student 
Gain displays). Data in the remaining displays (Bands, Graphs and Similar Schools) are available in 
school system dashboards, and are of less interest to parents than achievement and progress. An 
alternative, or supplementary, suggestion made by several senior school system officials and echoed 
in submissions was that a broader range of information should be made available over time, such as 
achievement in the arts or critical and creative thinking. 


The AEU submission was critical of the amount and quality of technical information on the My 
Schoo! website. Their submission noted the absence of information on NAPLAN test grading criteria, 
failure to disclose technical information on test equating procedures, and down-playing of the 
statistical error band in the Numbers NAPLAN page (p. 8). The submission particularly objected to 
the characterisation of schools using colour codes, when the data on which it is based includes a 
broad band of possible mean scores with a 95% confidence interval (p. 10). 


Other submissions suggested additional content be added to My School, including reporting on 
outcomes for different demographic groups and for students with a disability (ILSTE, p. 6). 


TRANSITION TO NAPLAN ONLINE 


Many submissions expressed concern about the whether fair comparisons were possible as schools 
transition to NAPLAN Online. The ISCA submission, for example, supported NAPLAN Online’s move 
to adaptive testing because it would increase the diagnostic value of NAPLAN but expressed concern 
about the appropriateness of school comparisons based on schools using either paper or online 
testing in 2018. Similarly, AISSA suggested that in the transition period, “statistical comparability of 
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both modes must be treated with sensitivity” (AISSA, p. 2) and reported survey evidence of low 
levels of confidence that paper and online test scores could be compared (p. 12-13). 


The AEU submission took a more definitive position, confirming that the union was opposed to the 
publication of the 2018 data, arguing that “the bi-modal nature of the data collection in 2018 breaks 
the longitudinal data series ...and no data for 2018 should be published publicly” (p. 12). 


Since submissions were received, however, the decision has been taken to publish the 2018 data on 
My School, managing the issue of comparison of results across test modes by using a dotted line on 
the Student Gain and Graphs displays and removing the Bands display for schools that completed 
NAPLAN Online. 


UNDERSTANDING STUDENT PROGRESS AND ACHIEVEMENT 


NAPLAN, MY SCHOOL AND SCHOOL IMPROVEMENT 


Judging by the effort that school systems have made to develop data dashboards to assist schools to 
interrogate their NAPLAN data and the degree to which NAPLAN targets have been included in 
schools’ annual plans and reports, school systems make a great deal of use of NAPLAN progress and 
achievement data in their school improvement strategies. In the independent sector, too, schools 
are reported as finding “NAPLAN student progress and achievement data useful to inform their 
school improvement strategies” (ISCA, p. 6). 


NAPLAN was, however, typically seen as “part of the bigger picture”, useful in high level planning 
rather than direct support for teaching. As one school system submission commented, “...data is 
only useful for long-term planning ... to identify broad strengths and weaknesses” (Townsville 
Catholic Education Diocese, p. 4). Schools reported that they look to NAPLAN for “trends in the 
cohort data” rather than “a focus on individual student performance” (confidential submission, p. 2). 
In this context, NAPLAN is often used “in conjunction with other information the school holds” (ACT, 
p. 4) and was seen as just one of many sources of data available to schools. 


Principals and parents’ groups often commented on the limitations of NAPLAN, characterising it as 
“one source of some useful information which is added to everything else a school might collect”, “a 
snapshot of achievement”, “a small part of the overall picture of a student’s learning journey”, “just 
one piece of the puzzle”, or “a moment in time”. As one school system senior officer put it, “it 


matters, but it is not the only thing that matters.” 


A little more than one-third of principals responding to the AHISA survey reported that they used the 
My School displays to compare their school’s result with other schools’ results. Among these users, 
the student gain by students with similar starting scores and student gain by schools with similar 
students were regarded as most useful (p. 15). More than half of AHISA principals reported using 
NAPLAN results in discussions with school boards, most often focusing on achievement in other 
independent schools in the area and student gain in all Australian schools (p. 15). This was 
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confirmed by one of the school system officials in the ISCA consultation meeeting, who said that 
NAPLAN results were “used at all the governors’ meetings | go to.” 


The AEU submission expressed concerns about the use of NAPLAN in school improvement strategies 
on the basis that higher NAPLAN scores were not necessarily a measure of success, especially if they 
were achieved by ignoring “the scope and depth of the curriculum” in favour of “drills and activities 
aimed at improving test performance” (p. 12). 


WHOLE-POPULATION ASSESSMENT AND SCHOOL IMPROVEMENT 


All of the government and Catholic school systems currently use whole-population NAPLAN data in 

their data analytics dashboards and school improvement strategies. They also use whole population 
data to identify issues for sub-populations, an essential equity measure that would not be available 
from sample testing.> 


Several of the school system authorities expressed concern about public reporting of school-level 
data rather than collection of whole-population data on which school-level data are calculated. The 
ACT government submission argued that data for school improvement was necessary but did not 
need to be made available on My School (p.4). Similarly, the Queensland submission drew attention 
to the joint statement on NAPLAN released by the Queensland Department of Education and the 
Queensland Teachers Union which acknowledged that “schools — through direct contact and their 
school websites — are best placed to provide up-to-date information about their context, their 
students, their wider community and the enriching learning experiences they offer to students and 
staff”.°? 


This kind of concern about school-level reporting was echoed by several state independent school 
authorities. AISSA acknowledged that NAPLAN population testing is “a valid and reliable research 
base for governments, schools and the wider community” but argued that “comparative reporting 
does not enhance this research base nor improve outcomes for students” (p. 3). Similarly, a 
confidential submission from this sector commented that “whole population data is useful but [we] 
feel that it should only be made available to government representatives, sectors and their 
representatives” (p. 6). 


Among the Catholic school jurisdictions providing submissions, the QCEC submission indicated that 
their school authorities use whole-population assessment for school improvement purposes, but 
they would support further consideration of the balance between cohort and sample testing. 
Similarly, Independent Schools Queensland noted that: 


Whole-population assessment can support schools to benchmark student and cohort 
progress and inform school improvement agendas, however ISQ encourages debate about 


52 This issue has recently been highlighted by Andy Hargreaves in his appearance before the Scottish National 
Standardised Assessments Inquiry (p. 5). 
https://www.parliament.scot/S5_Education/Meeting%20Papers/20190130ES Meeting papers.pdf 


53 https://www.qtu.asn.au/js-data 
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the value of moving to sample, rather than whole cohort testing in literacy and numeracy, to 
reduce the perception of NAPLAN as a high-stakes competitive testing regime (p. 6). 


Some principals’ associations were in favour of replacing NAPLAN with sample tests and others were 
not. Those who were in favour of population testing were often concerned about publication of 
school-level data. As a spokesman for one of the secondary principals’ associations explained: 


“NAPLAN has a lot of valuable data. It should be continued. It is a great tool for 
accountability within sectors, across sectors and for government accountability. Principals 
do not contest the need for accountability. The issue is who has access to the data. As soon 
as it becomes public it is used to rank schools.” 


AHISA principals were divided over whether whole population assessment was necessary to meet 
school systems’ and governments’ need for information to support school improvement, with about 
half (51%) agreeing that whole-population assessment was necessary. Fewer than half (35%) agreed 
that sample testing would be sufficient for this purpose. 


The diversity of opinion among principals was echoed among parents’ associations. Catholic School 
Parents Australia suggested that a suite of assessments made available to schools through ACARA, 
from which a sample of schools’ results could be examined for state and national accountability 
purposes. Several of the state affiliates of ACSSO were in favour of moving to sample testing, but 
APC’s view was that: 


“Most parents understand the value of whole-population assessment data and expect that it 
would drive changes in teaching and pedagogy to achieve growth for the students in their 
school, and at a system level” (p. 3). 


Continued collection of whole-population data was supported by three of the research and advocacy 
groups that provided submissions. The Grattan Institute submission opposed any move to replace 
NAPLAN with a national sample test. They argued that although teacher-generated data are 
essential for targeted teaching, assessment for accountability should be kept separate from 
assessment for teaching and learning, and that assessment for accountability requires whole- 
population assessment (pp. 3-4). The Smith Family submission supported continuation of whole- 
population data on the grounds that access to quality data is the foundation of improving students’ 
educational outcomes (p. 3). The CIS submission argued that there would be no transparency of 
outcomes at the school level without population testing and that systems, schools and governments 
would no longer be accountable for school-level performance (p. 7). The ILSTE submission supported 
a move to sample testing as a sufficient “health check” of system performance but noted that “if 
NAPLAN becomes a sample test, other mechanisms will need to be considered to enable ongoing 
monitoring of school performance” (p. 12). 


Both of the unions advocated for replacement of census tests by sample tests. The AEU submission 
argued that the NAPLAN results currently collected are unsuitable for anything other than state, 
national and demographic group trends, “and it is therefore unnecessary to test the entire school 
populations” (p. 12). The AEU’s preference is to move towards the approaches to assessment 
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championed in the Gonski Through Growth to Achievement report.™ Similarly, the IEU reported that 
two-thirds of their members supported a move to sample testing and overwhelmingly favoured the 
development of formative assessment tasks, available online as a method of assessing students 
against national benchmarks (p. 11). 


USING ACHIEVEMENT DATA TO INFORM TEACHING 


NAPLAN, MY SCHOOL AND TEACHING 


Many submissions distinguished between the use of NAPLAN data to inform teaching and the use of 
My School to access data for this purpose. The WA Education Department’s submission, for 
example, confirmed that: 


NAPLAN data is used by schools to identify trends in student achievement, measure 
individual growth and identify areas that need a greater focus ...[but] ... teachers and schools 
tend to access information regarding NAPLAN provided on the Department’s local system, 
rather than information from the My Schoo! website. (p. 3) 


Similar comments were made by independent and Catholic school authorities. ISCA, for example, 
balanced the observation that “independent schools find the NAPLAN student progress and 
achievement data useful” with the observation that “access to detailed analytical packages provides 
more nuanced information and the opportunity to triangulate results with other assessments” (p. 8). 
The Queensland Catholic Education Commission confirmed that teachers “do not generally use the 
My School website but rather access NAPLAN data through system data monitoring tools” (p. 5). 
About two-thirds of AHISA principals agreed that NAPLAN data are useful to inform class-level 
strategies or wider school improvement. Most also reported using it in triangulation with other 
internal or externally benchmarked assessments (p. 18). Beneath this broad agreement the AHISA 
survey reported a range of views about the value of NAPLAN. One principal reported that NAPLAN 


had been “incredibly useful”, allowing the school to “identify serious deficiencies in our learning 
program and to take decisive action” (p. 18). Another described NAPLAN data as “one dimensional”, 


noting that “it doesn’t make the data unimportant, just limited in scope and value” (p. 18). 


The AEU submission focused on members’ perception of the impact of NAPLAN on teachers and 
students, rather than the degree to which NAPLAN was used to inform teaching. According to the 
Union’s 2018 State of Our Schools Survey, about three-quarters of members agreed that they were 
under more pressure to improve NAPLAN results and that NAPLAN was increasingly used to measure 
school performance. About two-thirds agreed that there had been increased student stress in the 
leadup to the test and that there had been a greater focus on preparing for the test. The result, the 
AEU concluded, has been to “de-skill teachers, to stifle morale of the workforce and degrade the 
status of the profession” (p. 14). 


54 https://docs.education.gov.au/documents/through-growth-achievement-report-review-achieve- 


educational-excellence-australian-O 
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Schools use a wide variety of standardised and local assessment instruments to inform teaching. 
More than a dozen assessments were identified by principals responding to the AHISA survey (p. 19) 
and many more than a dozen in the IEU submission (pp. 6-8). Virtually every submission or 
consultation meeting that discussed assessments used by schools mentioned the ACER Progressive 
Achievement Tests in Reading (PAT-R) and Progressive Achievement Tests in Mathematics (PAT-M), 
which are provided to many schools through site licences held by school sectors and systems. 


Similarly, schools use a very wide range of approaches to managing NAPLAN data. According to the 
AHISA survey, most of their schools conduct some kind of digital analysis of NAPLAN data. The most 
common approaches were locally developed spreadsheet analyses (27%), software developed by 
public education systems (21%) and commercially available data analytics tools (20%). The 
remainder used software developed by the state or territory Association of Independent Schools or 
software developed internally or commercially to the school’s specifications (p. 20). By volume, the 
most commonly used data analytics platform is Scout, the data and analysis platform designed by 
the NSW Department of Education and available to government, Catholic and independent schools 
in the NSW and the ACT. 


In the decade since My Schoo! was launched, a large number of such NAPLAN data analysis packages 
have been developed by school systems and sectors, curriculum and assessment authorities and 
commercial providers. At least a dozen data analytics packages were mentioned in submissions or 
consultation meetings: 


e SunLANDA (Queensland Curriculum and Assessment Authority)°° 
e OneSchool (Queensland Department of Education)*° 

e DataPAK (Independent Schools Council of Queensland) 

e Scout (Department of Education, NSW)?” 

e CeD3 data warehouse (used by some Catholic school systems)°® 
e Panorama (Department of Education, Victoria) °° 

e Aspire (Department of Education, Tasmania) 

e School Improvement Dashboard (Department for Education, South Australia) 
e NAPLAN Analysis Tool (independent and Catholic schools, South Australia) 


e Student Achievement Information System (Education Department, Western Australia) © 


°° https://www.qcaa.qid.edu.au/p-10/naplan/test-reporting-analysis/sunlanda 

56 https://education.qld.gov.au/parents-and-carers/school-information/oneschool-qparents 

57 https://education.nsw.gov.au/about-us/educational-data/scout/resource-library/schools-dashboard 

58 https://ced3.catholic.edu.au/ced3-data.html 

5° https://www.education.vic.gov.au/school/teachers/management/improvement/Pages/panorama.aspx 

5° https://www.education.sa.gov.au/teaching/school-improvement 

51 https://www.ais.sa.edu.au/events/naplan-data-analysis-introducing-v-1-3-analytical-tool-paper-online-test- 
training-option-3/ 


62 http://det.wa.edu.au/accountability/detcms/navigation/school-performance/school-and-student- 


performance-data/ 
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e Brightpath (provided to WA schools through the School Curriculum and Assessment 
Authority)? 

e eDash (Department of Education Northern Territory) 

e Reporting and Analysis Achievement Data (Northern Territory Board of Studies) 


The ILSTE submission notes that the level of data analytics undertaken in schools is in some cases 
limited by access to appropriate software and school-based data literacy (p. 10). Reflecting this 
concern, several of the independent school sector authorities and principals’ associations in smaller 
jurisdictions mentioned that they would support the development of a comprehensive, common 
national school data dashboard. 


TIMELINESS OF NAPLAN REPORTING 


The move to NAPLAN Online was almost universally seen as the best opportunity to improve the 
timeliness of NAPLAN reporting, and there were few other comments on timeliness in submissions 
and consultations. Among organisations opposed to NAPLAN census testing, one suggested that 
timeliness could be improved because “a stratified sample should be able to be completed in a much 
timelier manner” (AEU, p. 15). 


REPORTING NAPLAN DATA TO PARENTS AND STUDENTS 


COMMUNICATION WITH PARENTS AND THE COMMUNITY 


Individual student reports are provided to parents of all children undertaking NAPLAN tests. Many 
stakeholders confirmed that these results are discussed with teachers in the context of parent- 
teacher meetings and that NAPLAN results may be mentioned in school newsletters. Principals’ 
associations reported that schools do not generally provide school-level NAPLAN results in detail to 
parents but might include a broad comment about progress or achievement in a newsletter or 
mention if they had been identified as an ACARA “high gain” school. The AHISA survey concluded 
that their schools are more likely to discuss students’ NAPLAN results in response to parents’ queries 
than at parent-teacher meetings, in writing or online on the school’s intranet. School staff, according 
to the Queensland NAPLAN review, indicated that NAPLAN offered little or no value in informing 
parents about their child’s progress. Both the QCEC and ISCA submissions noted, however, that their 
schools were more likely to report NAPLAN data in a locally contextualised way than to reference 
the My School website in their communication. 


My School is by no means the only public source of school-level NAPLAN data in Australian states 
and territories. The broadest public availability of school-level NAPLAN data is in Queensland, where 
NAPLAN outcomes for public, Catholic and independent schools are available online from the 


83 https://www.brightpath.com.au/ 
64 https://education.nt.gov.au/___data/assets/pdf_file/0009/542898/2017-NTBOS-Annual-Report.pdf 


55 The QCEC submission provides a list of analyses that could be provided by a new national data analytics tool 
(p. 6) 
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Queensland Curriculum and Assessment Authority website.® In several jurisdictions, the 
government school system provides a searchable website that includes school-level summaries of 
NAPLAN data. In Western Australia, Schools Online provides average NAPLAN scores and the 
proportion of students at, above and below the National Minimum Standard for all schools.® In the 
Northern Territory a searchable database®™ provides links to schools’ Annual Performance Report to 
the School Community which include NAPLAN achievement tables. 


More commonly, annual reports containing NAPLAN data are available on school websites. In South 
Australia, for example, annual school reports on government schools’ websites use a standardised 
format including proportions of students achieving NAPLAN proficiency levels, the proportion of 
students in three NAPLAN progress bands and the proportion achieving in the upper two bands.’° In 
the ACT, annual school board reports on government school websites include a table of NAPLAN 
mean scores and an annual action plan detailing progress against NAPLAN targets.” In Victorian 
public schools, annual implementation plans on government school websites include quantitative 
NAPLAN improvement targets but do not include tables of annual NAPLAN results.’? In NSW, annual 
reports appear on government school websites but do not publish NAPLAN results in a common 
format.’? In Tasmania alone, among all of the government schools systems, individual school-level 
NAPLAN data do not appear to be available to parents or others through school or system websites, 
except through the My Schoo! website. 


The quality of communication materials and approaches designed for parents and the wider 
community was raised in several submissions. The Smith Family, for example, noted that it was 
difficult for parents to use NAPLAN information understand their children’s progress. They also 
noted that, given the level of discussion of NAPLAN in the community, “there is also a need for a 
wider group of stakeholders to be better informed on NAPLAN” (p. 6). Quality of communications 
was also a major focus of the Queensland Government’s submission, reporting that the Government 
will implementing a targeted communications strategy concerning the purposes and appropriate 
uses of NAPLAN (p. 3). 


The APC, The Smith Family and the QCEC submissions raised the possibility of improving feedback to 
parents about students’ progress over time, from Years 3-9, on a single report, with several noting 
that this would require a national unique student identifier. 


56 https://www.qcaa.qid.edu.au/downloads/publications/qcaa_stats_naplan_18 outcomes.pdf 
87 https://www.det.wa.edu.au/schoolsonline/home.do 
88 https://education.nt.gov.au/education/publications/government-school-annual-reports 


5° For example, 
http://web.ntschools.net/w/antgsr/Annual%20Reports/Casuarina%20Street%20PS_ 2017%20Annual%20Perfor 


man%2O0Report.pdf 
70 For example, https://www.wgs.sa.edu.au/uploads/files/Annual%20Report%202017.pdf 


7 For example, http://www.blch.act.edu.au/our_school/school_board and 


http://www.maribyrnongps.act.edu.au/our_community/school_ board 


” For example, http://mentoneps.vic.edu.au/wp-content/uploads/2018/08/2018-Annual-Implementation- 


Plan.pdf 
73 In NSW, construction of comparative league tables from these school reports would be contrary to the NSW 


Education Act 1990. 
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NAPLAN, MY SCHOOL AND SCHOOL CHOICE 


Submissions and stakeholder conversations on parents’ use of the NAPLAN data on My Schoo! most 
often focused on the issue of school choice. Parents from one government school jurisdiction 
commented that there was little value in using My School data for school choice purposes, because 
enrolment in their jurisdiction was controlled by schools’ residential boundaries. Others mentioned 
that use of My School to inform school choice was more likely when children were moving from 
primary to secondary school, or when families were relocating. Defence families, who move 
relatively often, were thought to be more likely to use My School in choosing schools. In the absence 
of these reasons to move or choose schools, the Grattan Institute submission cites empirical 
evidence that “In Australia, families don’t generally move to high-performing schools nor leave low- 
performing ones” (p. 6).”* 


Survey data included in the submissions suggested that relatively few parents use My School to 
inform school choice. The APC submission reported that according to their 2018 survey, “Only 5% of 
parents in APC’s 2018 national survey reported using NAPLAN results when choosing a school for 
their child.” A somewhat higher estimate is provided by ISQ’s 2018 What Parents Want”? survey 
which reported that only 18% of parents identified My Schoo! among their top three sources of 
information. Much more common sources of information were family, friends and colleagues; other 
parents with children at the school; school open days; school websites and social media; and family 
members attending the school. 


The ISQ survey also reported that when parents used My School, they were more likely to look at the 
school profile pages than the NAPLAN data pages. This conclusion is consistent with ACARA’s 
internal Google Analytics analysis of the My School website, which shows that the school profile 
pages are the most frequently visited; only 20% of My School page views are of NAPLAN data pages. 
Several submissions mentioned a recent study of parents’ use of the My School website conducted 
by Colmar Brunton for ACARA in 2018.”° This large-scale parent survey reported that 44% of parents 
would use the My Schoo! website to see how their child’s school is performing and 39% would use it 
to help inform decisions for their child’s education. Three-quarters of all parents (74%) believe that 
it is important for parents to access the My Schoo! website and the information it provides on 
schools. Only one in eight (13%) parents responding to the survey believe it is not important. While 
only one-third of parents (34%) had used the My School website, nearly nine in ten of these users 
rated it as being at least quite important. 


STUDENTS AND NAPLAN REPORTING 


Despite the Issues Paper’s invitation to comment on information about NAPLAN that students might 
need to contribute to their own education, it was mentioned in only a few submissions and 
stakeholder interviews. 


7 https://grattan.edu.au/wp-content/uploads/2014/04/myth_of_ markets _in_school_education.pdf 
75 https://www.isq.qld.edu.au/advocacy-representation/what-parents-want-survey 

76 https://acaraweb.blob.core.windows.net/acaraweb/docs/default-source/assessment-and-reporting- 
publications/2018-naplan-online-parent-research.pdf?sfvrsn=2 
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One parent submission noted that reporting of NAPLAN information to students “six months after 
testing takes place is highly unlikely to be of any benefit to their education” and another suggested 
that younger students should only be given NAPLAN performance information in consultation with 
their parents. Several submissions suggested that the move to online testing might reduce the delay 
and make individual student NAPLAN reports more useful for students. The ILSTE submission cited 
research confirming that there had been little engagement of students about NAPLAN, and that this 
was a concern in view of other research that demonstrates the impact of students’ personal agency 
in enhancing self-regulation and their own learning. The submission questioned, however, whether 
this was possible with the current NAPLAN tests since “NAPLAN’s diagnostic and formative 
assessment potential exist at the class or group level, identifying areas of instruction where student 
response patters indicate a consistent gap” (p. 13). 


Among the suggestions for improved communication with students, IST and ISCA recommended that 
students receive age appropriate summaries of their NAPLAN results. ISQ noted that feedback 
focused on a student’s gain a may provide a better link between test scores and understanding of 
their own educational progress, but that “optional formative assessment tools aligned to learning 
progressions ... may provide a better opportunity for students to engage in conversations about their 
educational progress” (p. 9). Similarly, AISSA noted that “the power of data is in the ability to tell a 
story”, that use of growth data over time supports students to make decisions, but that the NAPLAN 
data used for this purpose does not come from the My School website (p. 4). The Queensland 
Government submission indicated that it intended to develop a targeted communications strategy 
to help students (and others) to understand the purposes and appropriate uses on NAPLAN data (p. 
3). 


SUMMARY 


PERCEPTIONS OF MY SCHOOL AND NAPLAN REPORTING 


‘The right to high quality information and the risk of misuse 


There is no national consensus on the balance between the community’s right to high quality 
information and the risk of misuse on NAPLAN data. 


The Australian Government and most state and territory governments were broadly satisfied with 
the current balance, citing the principles of transparency and accountability. The state and territory 
governments that were more cautious about the current balance defended the benefits of 
accountability but cited concerns about narrowing the curriculum, teaching to the test, student and 
teacher wellbeing and the use of My School as a resource for school choice. 


Non-government school systems and sectors supported the continued publication of school-level 


NAPLAN data but shared the concerns raised by some state and territory governments. In addition, 
many non-government school sector stakeholders expressed concern about ICSEA-based “similar 
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school” school comparisons and the narrowness of My School’s NAPLAN-based reporting of school 
performance. 


Feedback from parent stakeholders was mixed. Several stakeholders argued that the current level of 
transparency was appropriate, but some of these also expressed concern that My School and 
NAPLAN had skewed parents’ understanding about quality of schooling. 


Views of principals’ associations were also mixed. Most associations recognised the desire of 
governments to be transparent with the data and accountable to the community, but most were 
also concerned with the consequences of public access to school-level data. Negative consequences 
cited included ranking of schools, and the narrowness of NAPLAN-based judgements about school 
quality. 


Union responses to the review supported public availability of data to track progress at the systemic 
level, but either entirely rejected reporting of school-level NAPLAN test results or argued for 
mitigation of negative impacts on teaching, well-being and enrolment behaviour. 


Several of research and advocacy groups argued that publication of NAPLAN data is essential, either 
for transparency, accountability or school improvement. Others argued that publication of NAPLAN 
on My School! did not achieve an appropriate balance between the right to know and the possibility 
of misuse, either because of technical limitations to the tests or evidence cited about the negative 
impact on teaching and well-being. Opportunities for improvement identified by research and 
advocacy groups included broadening the range of assessments to include students’ general 
capabilities, focusing more on learning gain and alignment with learning progressions, changing the 
scope of the literacy and numeracy assessments or moving NAPLAN from census testing to sample 
testing. 


Statistically similar schools 


The use of ICSEA-based similar school calculations was deeply unpopular, especially with non- 
government system and sector stakeholders and school principals. Some of the objections were 
technical, with concerns expressed about self-reported data, the uneven impact of home language 
and culture on ICSEA scores, or the absence of other explanatory variables such as gender and 
disability. Other concerns were more fundamental, with stakeholders contesting whether statistical 
similarity had anything to do with the actual similarity of schools across geographical locations, with 
cultural contexts or levels of financial resources. Some stakeholders suggested that ICSEA be revised; 
others suggested that the focus should be on student gain compared with students who had the 
same starting scores. 


Commentary on the quality of the explanatory materials on similar schools was mixed. Several 
stakeholders, including one parent group, rated the concept of statistically similar schools as well 
understood, but other parent and sector groups rated the information as not easy to understand or 
hard to find. 
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Information included on My School 


Although stakeholders’ views about ease of understanding the language and graphics on My School 
ranged from “easy” to “requiring a high level of data literacy”, there was broad agreement that 
there is too much emphasis on NAPLAN on the My School website. Broader views of schooling, 
including student capacities beyond literacy and numeracy would be welcomed by many. 


Transition to NAPLAN Online 


There was some scepticism, especially from independent school and teachers’ union stakeholders, 
about the comparability of pencil-and-paper and online NAPLAN results. The 2019 My School 
website NAPLAN displays have responded to these concerns by using dotted lines in the Student 
Gain and Graphs displays for schools that changed mode of assessment, and removing the Bands 
displays for NAPLAN Online schools. 


UNDERSTANDING STUDENT PROGRESS AND ACHIEVEMENT 


NAPLAN, My School and school improvement 


School systems and sectors have made substantial investments in data analytics platforms designed 
(among other things) to assist schools to use their NAPLAN data for school improvement planning 
and feedback. The quality of these systems, as well as the degree to which they are integrated into 
system and sector reporting arrangements, mean that although NAPLAN data are widely used My 
School is rarely used by schools for school improvement purposes. When it was used, principals 
preferred the displays that focused on student gain to those that focused on average scores or 
school comparisons. 


Stakeholders often commented on the limitations of NAPLAN for school improvement, 
characterising it as just a snapshot, or a piece of the puzzle, one among many areas of interest to 
schools in their school improvement planning. 


-Whole-population assessment and school improvement 


All of the government and Catholic school systems and most independent schools currently use 
whole-population NAPLAN data in their data analytics, school review and school board reports. 
There were, however, concerns about the impact of publication of school-level whole-population 
data. For this reason, there was some interest in the prospect of moving NAPLAN from whole 
population to sample testing. The strength of this interest in sample testing ranged from openness 
to a debate, at one extreme, to vigorous advocacy by unions and the Gonski Institute. Others argued 
that it was important to continue whole-population testing for accountability and equity purposes. 
As one of the research group stakeholders noted, if NAPLAN were to become a sample test, school 
systems would most likely adopt other whole-population assessments to monitor school 
performance. 
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USING ACHIEVEMENT DATA TO INFORM TEACHING 


NAPLAN, My School and teaching 


There was little evidence that the NAPLAN displays on My School influenced day-to-day teaching and 
learning in schools. Much more common was the use of NAPLAN data to identify trends in student 
achievement at the school level, often used in triangulation with other standardised and local 
achievement data. 


Assessment tools and data analytics 


More than a dozen data analytics platforms were mentioned by stakeholders, and they were used in 
virtually all school sectors and systems to understand item-level student performance and identify 
strengths and weaknesses in teaching programs. Some platforms, such as the NSW Scout and 
Catholic schools’ CeD3 were used in multiple jurisdictions; others such as the WA SAIS system were 
restricted to a single government school system. Some independent schools used these or other 
commercially available software; others used locally developed spreadsheet-based systems. 


Timeliness of NAPLAN reporting 


The move to NAPLAN Online was seen by most as the best opportunity to improve the timeliness of 
NAPLAN reporting. 


HOW MY SCHOOL AND NAPLAN ARE REPORTED TO STUDENTS AND PARENTS 


Communication with parents and the community 


Individual student reports are provided to all parents of students who take NAPLAN tests. 
Stakeholders reported that schools sometimes discussed these with parents, especially if parents 
sought more information. Some stakeholders dismissed NAPLAN as a source of information to 
parents, preferring more contextualised information about student progress. School-level NAPLAN 
results are sometimes mentioned in school newsletters and almost universally used in school annual 
reports, improvement plans and reports to school boards. 


Although My School is the only public, national source of school-level NAPLAN data, school-level 
results are very widely available in many jurisdictions. One state publishes them for all government 
and independent schools on its curriculum and assessment authority website; several others provide 
a searchable data base for government schools; and most of the remaining states and territories 
require government schools to publish annual reports containing standard NAPLAN displays on their 
school websites. 


The quality of communications with parents and the community was raised in several submissions, 
and at least one state is considering how the unintended consequences of NAPLAN testing may be 
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ameliorated by clear and more effective communication about the purposes and proper uses of 
NAPLAN data. 


NAPLAN, My School and school choice 


The impact of NAPLAN and My School on school choice and competition between schools was an 
important theme in stakeholder submissions and conversations. The evidence is that relatively few 
parents use My School! to inform school choice, and that those who do look at My School are more 
likely to look at the school profile page than the NAPLAN data pages. Nor do parents generally 
regard NAPLAN performance as an important source of information when choosing a school. There 
is, however, some evidence that parents believe that it is important to have access to this 
information. This view is shared by some, but not all, of the parent stakeholder groups consulted in 
this Review. 


Students and NAPLAN reporting 


Few stakeholders took up the invitation to comment on how information on NAPLAN achievement 
might better be communicated to students. Among those who did, however, suggestions included 
provision of age-appropriate summaries and a reminder of how important timely information about 
achievement could be in enhancing students’ self-regulation and their own learning. 
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CHAPTER 4: FINDINGS AND RECOMMENDATIONS 


The NAPLAN Reporting Review has been framed by the four Terms of Reference: 
1. Perceptions of NAPLAN and My Schoo! data, including the potential for misinterpretation or 
misuse of data; 
2. How My School and NAPLAN reporting contribute to understanding of student progress and 
achievement; 
3. How schools use achievement data, including NAPLAN, to inform teaching; and 
4. How My School and NAPLAN data are reported to students and parents. 


In this final Findings chapter, Terms of Reference 2, 3 and 4 are considered before Term of 
Reference 1. The purpose of this re-ordering is to provide findings about patterns of use of NAPLAN 
and My School by schools, systems and sectors, parents and students, before returning to broader 
perceptions about whether these patterns of use reflect an appropriate balance between the right 
to high quality information and the possibility of misunderstanding or misuse of data. 


FINDINGS 


UNDERSTANDING STUDENT PROGRESS AND ACHIEVEMENT 


NAPLAN was introduced in 2008 and NAPLAN data have been available on the My School website 
since 2010. In that decade, NAPLAN has become a fundamental part of the school improvement 
process in most school systems and sectors. NAPLAN’s role in supporting accountability and 
transparency was valued by most school sectors and systems as well as many principal and parent 
stakeholder groups. It remains the only authoritative source of whole-population literacy and 
numeracy data and is used by governments and school systems to make judgements about school 
and school system performance. Acknowledging this role, some governments have reservations 
about unintended impacts of NAPLAN testing and the publication of NAPLAN data on My School. 


Finding 1. Australian governments and school systems rely on NAPLAN to make judgements 
about school and school system performance, but some have reservations about unintended 
consequences of NAPLAN testing and the publication of NAPLAN data on My School. 


NAPLAN data have been integrated into the data analytics platforms that have been developed by 
school sectors and systems in recent years. School leaders and teachers participating in focus group 
interviews confirmed that NAPLAN data are widely used for school improvement, accountability and 
identifying trends and gaps in achievement. Schools use their school system or sector data analytics 
platforms or locally developed data strategies rather than My School for this purpose. 


Finding 2. School sector and system data analytics platforms are widely used in 


understanding student progress and achievement, but schools do not use My Schoo! data 
displays for this purpose. 
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Although stakeholders and focus group participants confirmed that NAPLAN data are widely used in 
identifying trends and gaps in achievement, in school planning and in annual reporting to school 
authorities, many were concerned about aspects of NAPLAN. Some were concerned by what they 
called the narrowness of NAPLAN’s focus on literacy and numeracy, arguing that there is much more 
to school quality than NAPLAN results. Others were concerned that NAPLAN results are just one 
point in time, or that there are long time-lags between testing and reporting. The consensus was 
that although there is value in NAPLAN, it is just one part of the puzzle for schools focused on 
improving teaching and learning. 


Finding 3. NAPLAN provides important but incomplete information on school quality. 


School staff in the focus groups were more likely to agree that NAPLAN data were useful for school 
improvement and accountability than for setting targets or making school comparisons. They found 
the Similar Schools displays on My School less useful than the others. Among stakeholders, there 
was a strong preference for data displays that focus on student gain and significant concern about 
the use of statistically similar school comparisons. 


Finding 4. Data displays that focus on gain in student achievement were preferred to those 
that used ICSEA-based calculations to compare statistically similar schools. 


USING ACHIEVEMENT DATA TO INFORM TEACHING 


Although every one of the sample schools demonstrated a deep understanding of their students’ 
achievement data and actively used it for school improvement purposes, NAPLAN was rarely used to 
inform day-to-day teaching practice. Where it was used, it was in triangulation with other 
assessments. Schools had available to them a wide variety of standardised and local assessment 
instruments and were more likely to use these to inform teaching. Dozens of such tests were 
mentioned by stakeholders, in sample schools and in submissions. Virtually every stakeholder 
conversation that touched on assessments used in schools mentioned ACER’s PAT-R and PAT-M 
tests. 


Finding 5. Schools triangulate NAPLAN data with a wide range of other assessments 
including large-scale nationally normed and standardised tests. 


REPORTING NAPLAN DATA TO PARENTS AND STUDENTS 


There have been significant changes in the international context for public reporting of school-level 
achievement data in recent years. The United States has moved from national to state-based 
reporting procedures. Hong Kong has moved from whole-cohort testing to sample testing in Year 3. 
Singapore is moving from scale scores to reporting only in achievement bands. Scotland has made 
national standardised testing available to schools but publishes detailed school-level results only in 
broad bands based on teacher judgement. Wales grades schools but no longer publishes school-level 
results. New Zealand no longer has whole population testing; school reviews are published but do 
not require reference to national or standardised assessments. There is, however, no obvious 
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empirical relationship between levels of transparency of school-level achievement data and 
educational achievement among these jurisdictions.. 


Internationally, the two most common approaches to public reporting school-level assessment data 
are: 

e achievement, and 

e impact on achievement. 


Achievement is sometimes reported in terms of benchmarks, bands or averages. Tennessee, 
California, New York City and Ontario, for example, report the proportion of students reaching a 
benchmark. Arizona and England report the proportion in achievement bands. Singapore reports the 
lowest, highest and average scores in the national primary school assessment. My School provides 
averages as well as the proportion achieving in each band. 


Impact on achievement is reported using value-added, student gain or similar school/similar student 
calculations. Value-added calculations are published in Tennessee and were used in England for 
many years. Measures of student achievement gain are used in Arizona, in England and on the My 
Schoo! website. Statistically similar student calculations are published in the New York City school 
district and similar school calculations are published on the My Schoo! website. 


Although most jurisdictions publish achievement data using simple numerical scores in individual 
tests, some create composite indicators of achievement or impact and represent these in terms of 
overall letter or numerical grades, colour codes or graphics. My Schoo! does not simplify data by 
using composite indicators. 


Finding 6. The My School website provides more comprehensive and detailed school-level 
performance data displays than any of the international jurisdictions considered in this 
Review, but less information than is provided in Australian system and sector data analytics 
platforms. 


My School is the only publicly available source of school-level NAPLAN data on every Australian 
school. In many jurisdictions, however, governments make school-level data available to the public, 
either through searchable online databases, or through standardised annual reports or improvement 
plans posted on departmental or school websites. Independent schools also disclose their NAPLAN 
results on their websites in school board papers and annual reports. 


Finding 7. School-level NAPLAN results are widely but not universally available from sources 
other than My School. 


Individual NAPLAN results are provided to parents, but stakeholders confirmed that discussions 
about individual results were more likely when parents actively sought more information from 
schools or teachers. Schools preferred to use more contextual, and often timelier, achievement data 
than NAPLAN in discussions with parents. School-level NAPLAN results are sometimes reported in 
school newsletters and about two-thirds of the focus group parents recalled their school providing 
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such feedback about NAPLAN. Parent stakeholder groups reported that such results have a relatively 
low profile in the schools. 


Finding 8. Schools make limited use of school-level NAPLAN data in communication with 
families and prefer timelier and more contextual data when discussing individual student 
achievement. 


Many stakeholders were concerned that publication of NAPLAN data on My School had increased 
competition between schools, led to unfair comparisons or prompted parents to choose schools on 
the narrow basis of NAPLAN results. Although parents participating in the focus groups reported that 
the My School NAPLAN displays could be useful for choosing schools, fewer than half had looked at 
My School before the focus group meetings and few believed that My Schoo! was often used for this 
purpose. Almost all focus group parents agreed that school-level NAPLAN results should be available 
on a public website. Similar conclusions about parents’ perceptions have been reported in the 
recent Colmar Brunton review’’ commissioned by ACARA: 


“Parents had generally not given this issue prior thought and, when questioned, found it 
difficult to see a reason why the information on the My School website would not be freely 
available in the public domain” (p. 5). 


Few focus group teachers or school leaders had personal experience of parents choosing or leaving 
schools because of NAPLAN results, and fewer agreed that the NAPLAN pages on My School were 
useful in supporting parents’ school choice. Stakeholders confirmed that NAPLAN results are not 
among the most important factors in parents’ school choices. 


Stakeholder consultation with parent groups, focus group interviews and research conducted for 
ACARA confirmed that parents often — but by no means universally — expect that school-level results 
should nevertheless be made public for transparency and accountability. 


Finding 9. Parents generally do not place a high priority on NAPLAN results when choosing a 
school, but many believe that transparency and accountability require publication of these 
results. 


Stakeholders varied in their judgements about the accessibility of My School’s NAPLAN pages. 
Considering the range of concerns raised by parent stakeholders about the purposes for which the 
website was designed compared with the purposes for which it is now used, there is merit in 
considering a broader communication strategy that clarifies appropriate uses and purposes for My 
School and NAPLAN. 


Finding 10. My Schoo! and NAPLAN would be strengthened by a public communication 
program that clarified the purposes and proper uses of the data and the website. 


ve http://docs.acara.edu.au/resources/20150304 Perspectives on the My School Website.pdf 
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Few stakeholders, submissions or focus group members commented on communication to students 
about NAPLAN. In the student focus groups about one-third of students recalled their teachers 
providing any feedback on class or school NAPLAN results. Those who could, recalled their teacher 
going over the questions after the test or acknowledging good performance. The vast majority of 
students participating in the focus groups, however, were aware of the importance to their families, 
teachers, school (and indeed themselves) of their NAPLAN results. Considering the research 
evidence on the impact of appropriate information on students’ self-regulation and achievement, 
NAPLAN would be strengthened by age-appropriate communication strategy directed at individual 
students. 


Finding 11. Students would benefit from age-appropriate explanations of their NAPLAN 
results. 


PERCEPTIONS OF MY SCHOOL AND NAPLAN REPORTING 


School systems and sectors rely on NAPLAN for accountability purposes and these purposes were 
accepted by a large majority of respondents to this review. Submissions were often framed, 
however, by stakeholders’ restatements of their understanding of the purposes of either NAPLAN or 
My School. 


Stakeholders’ understanding of the purposes of NAPLAN of ranged from system accountability to 
guiding teachers about concepts not yet attained by students. The Colmar Brunton report prepared 
for ACARA found the same spread of views, with parents suggesting that NAPLAN and My School 
were intended to facilitate comparisons among states, comparisons among local schools, for 
monitoring teacher or student cohort performance, or for monitoring the performance of individual 
students. They found a similar range of views about the uses of NAPLAN, including allocation of 
funds, school choice, school improvement planning and intervention with individual students, as did 
the Queensland Government’s recent NAPLAN Review. 7 


Finding 12. There is no settled view of the purposes or proper uses of NAPLAN and the My 
School website. Further clarification and communication of these issues would be 
welcomed. 


Some school principal stakeholders and the teachers’ unions contested whether NAPLAN provided 
valuable or rigorous data, but this view was not widely held among other stakeholders. What was 
more common, especially among principal stakeholder groups, was discomfort with the public 
release of school-level data. Most did not contest the collection of population data but did contest 
the value of making it public beyond the school and school system. As one principals’ association 
commented, “as soon as it becomes public it is used to rank schools”. 


Finding 13. Many stakeholders were concerned that publishing school-level NAPLAN data 
had made the tests “high stakes”, and that any negative consequences flowed from 


report.pdf 
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publication of NAPLAN data rather than the collection of data or provision of data to schools 
and school systems. 


Among stakeholder groups that reported negative consequences of NAPLAN and My School, three 
issues were most commonly raised: inappropriate school comparisons; diversion of teaching time; 
and stress and wellbeing. These negative consequences were predicted by opponents of population 


18° and 


testing long before NAPLAN and My School were established’’, have been reported in loca 
international®? studies of the impact of population testing, and were raised in the recent Queensland 
NAPLAN Review. The strength of concern about these issues, however, clearly varies among 
stakeholders, with views marked by regional differences, roles within or outside schools, ideological 


perspectives and the sources of empirical data to support or contest these claims. 


There are several dimensions to concerns about school comparisons. Some of the concerns were 
with the ICSEA-based comparisons on My School. Technical issues were raised, with stakeholders 
drawing attention to the calculation of predictive factors in ICSEA, such as home language 
background, disability or gender.*? Other stakeholders were concerned that statistically similar 
schools based on ICSEA calculations had little to do with actual similarity across geographical 
locations, cultural contexts and levels of financial resources. Most parents, teachers and school 
leaders participating in focus groups for this Review and some of the principal stakeholder 
organisations, however, agreed that colour-coding of cells and diamonds in My School NAPLAN 
displays — which are based on ICSEA similar school comparisons — was useful in interpreting the 
results. Such colour-coding to aid interpretation is very common among international jurisdictions 
that report school-level test results. 


Finding 14. Colour-coding of NAPLAN results was regarded as useful by many focus group 
participants, but stakeholders had concerns about the use of current ICSEA comparisons as a 
basis for the similar school calculations that underpin the colour-coding. A technical review 
of ICSEA would be well regarded. 


Internationally, strategies other than similar school calculations have been developed to estimate 
the impact of schools on students’ performance. Tennessee pioneered the use of contextual value- 
added calculations that were used in the United Kingdom for many years but have recently been 
replaced by measures of student achievement gain. The New York City school district makes 
comparisons based on gain achieved by a matched group of students with similar starting scores and 


7° Cobbald, T. (2004). Reporting on school performance. A paper prepared for the ACT Minister or Education 
and Training. Government Schools Education Council ACT. June 2004. 


http://www.saveourschools.com.au/file_download/16 
80 For asummary, see: Wyn, J., Turnbull, M., & Grimshaw, L. (2014). The experience of education: The impact of 
high stakes testing on school students and their families. 


81 See, for example, https://www.nfer.ac.uk/what-impact-does-accountability-have-on-curriculum-standards- 


and-engagement-in-education/ 


82 The most recent technical review of ICSEA was published in 2014: 


https://www.myschool.edu.au/media/1032/icsea_ 2013 generation report.pdf 
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demographic characteristics. Among teachers and school leaders participating in this Review’s focus 
groups, measures of gain were much preferred to statistical similar schools. Principals’ groups, 
independent and Catholic education authorities and research and advocacy groups were among 
those who advocated for a focus on gain in NAPLAN and My School displays. The consensus among 
state and territory government authorities was that displays on My School could be reduced to 
school achievement and student gain, as other estimates of school performance of interest to 
schools and school systems were available to schools from their data analytics platforms. 


Finding 15. Measures of student achievement and gain may be sufficient information for 
public accountability and transparency purposes. 


League tables have emerged in every international jurisdiction considered in the environmental 
scan, except New Zealand, including those in which governments already publish detailed and web- 
searchable school-level performance data. These league tables often involve composite indicators 
and gradings, simpler than and different from the information provided on official websites. 


In Australia, the data required to construct NAPLAN league tables are readily available in many 
jurisdictions: in a downloadable table on a government website in one state; in searchable data 
bases on government websites in several others; and in annual reports or improvement plans 
published on individual school websites in most other government school systems. 


My School currently seems to have a limited role in the propagation of school league tables, 
although it may have been more common in the past. Use of the My Schoo! data for this purpose is 
contrary to the terms of use of the website and ACARA has been active in pursuit of users who have 
ignored the terms of use. Two of the three recent league tables published in Australian newspapers 
have drawn on publicly available sources other than My School, and one appears to have drawn 
directly on My School NAPLAN data. 


Finding 16. Some, but not all, of the recent Australian league tables have drawn their 
NAPLAN data from publicly available sources other than My School. 


Judgments about negative consequences of NAPLAN varied by stakeholder group and data source. 
Most school leaders, teachers and parents consulted in this Review’s focus groups disagreed that the 
curriculum had been narrowed to boost NAPLAN results in their schools, but some suggested that it 
might be happening at schools other than their own. Most also disagreed that too much time had 
been spent on test preparation. In contrast, more than 70% of members responding to the IEU 
survey agreed that time spent on test preparation had impacted on time available for teaching the 
curriculum. According to the Queensland NAPLAN Review school staff reported that attention to 
NAPLAN and NAPLAN outcomes affected the full implementation of the Australian Curriculum, even 
though the policy discourse had given primacy to teaching the full curriculum and using school 
assessments (p.13). 


Finding 17. Concerns about the impact of NAPLAN on teaching and learning programs were 


reported but estimates of the severity of this impact varied among stakeholders and data 
sources. 
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Evidence of impact on staff and student wellbeing was also mixed. A 2017 study®? cited by the AEU 
reported negative impacts on student wellbeing and staff morale. Similarly, school staff responding 
to the survey conducted for the Queensland NAPLAN Review indicated that NAPLAN was having a 
negative effect on their well-being but their interview and focus group respondents were more likely 
to attribute this to media representations of NAPLAN or background morale issues than NAPLAN 
testing (p. 12). Students consulted in the Queensland survey did not agree that NAPLAN testing had 
a negative impact on their well-being. In the focus groups conducted for this Review, few parents 
agreed that their children and been too stressed by NAPLAN tests, but it was a cause for concern 
among about half of focus group teachers and leaders. 


Finding 18. Concerns about the impact of NAPLAN on wellbeing of teachers and students 
were reported but estimates of the severity of this impact varied among stakeholders and 
data sources. 


Many stakeholders canvassed alternative possibilities for the future of NAPLAN. Some suggested 
that it be replaced by online, formative assessments based on learning progressions; others 
suggested that the system-level accountability functions of NAPLAN could be met by sample testing. 
Neither of these proposals is within the scope of this Review, but both are relevant to considerations 
about current representation of NAPLAN in My School. 


The source of many stakeholders’ conviction that NAPLAN would be insufficient for the future was 
their interest in or support for the kind of assessment that the 2018 Gonski Through Growth to 
Achievement*™ report foreshadowed: 


“an online, formative assessment tool to help diagnose a student's current level of 
knowledge, skill and understanding, to identify the next steps in learning to achieve the next 
stage in growth, and to track student progress over time against a typical development 
trajectory” (p. x). 


The development of such a tool is some way off, although there are some current initiatives that 
would contribute to this work (ACER’s Progressive Achievement Tests®°, the ACARA Learning 
Progressions® and Brightpath®’, to name a few). It is unclear, however, whether such a tool would 
be the best approach to meeting the accountability purposes for which NAPLAN is currently used. 
The Grattan Institute submission warned that different assessment tools are appropriate at different 
levels of education systems: teacher-generated data are most appropriate for targeting teaching and 
standardised tests are most useful for assessing school and system performance. This is a complex 


stakes-testing-on-school-students-and-their-families 
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85 https://www.acer.org/au/pat 


86 httos://www.australiancurriculum.edu.au/resources/national-literacy-and-numeracy-learning- 
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trade-off and may not be able to be resolved until the proposed formative assessments have been 
developed. 


Finding 19. There is widespread interest in the development of on-line, formative 
assessments based on learning progressions. 


Some stakeholders have also advocated a move from whole-population to sample testing. This has 
been seen as an opportunity to reduce the reported negative effects of NAPLAN. Some school 
sectors have indicated an interest in contemplating this option. Other stakeholders, the teachers’ 
unions and the Gonski Institute particularly, have strongly advocated for it. Several of the research 
and advocacy groups have taken an alternative view, arguing that whole-population testing is 
essential for accountability and equity purposes. Whether or not sample testing would be sufficient 
for accountability purposes, and whether governments would agree to a move to sample testing, is 
uncertain. What is certain, however, is that all of the government and Catholic school systems and 
most independent schools currently make substantial use of school-level data that require 
population testing. 


Finding 20. Some stakeholders advocated for a national accountability system based on 
sample testing, but this is inconsistent with school systems’ and sectors’ current use of 
population NAPLAN data in their data analytics, school review systems and school board 
reports. 


RECOMMENDATIONS 


My School, NAPLAN and indicators of school quality 


Australian governments and school systems currently rely on NAPLAN to make judgements about 
school and system performance and most governments expect that data collected through a 
national assessment system should continue to be available to the public (Finding 1). Although the 
NAPLAN displays available on My School are not among the most important sources of information 
for parents when choosing their children’s schools, most parents also expect that school-level 
achievement data will be available to the public (Finding 9). 


My School’s NAPLAN displays are more comprehensive and detailed than school-level performance 
displays available to the public in comparable international jurisdictions (Finding 6). NAPLAN, 
however, provides incomplete information on school quality (Finding 3). Many would welcome less 
emphasis of NAPLAN data on the My Schoo! website. One option is to reduce the content of My 
School, removing the least used and understood NAPLAN data pages. Another is to reduce the 
number of NAPLAN assessments published on My School, perhaps focusing on the core assessments 
of reading, writing and numeracy and removing the spelling and grammar results (although these 
assessments could continue to appear on school system and sector data analytics platforms). 


Recommendation 1: That the number of NAPLAN displays on My School be reduced. 
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NAPLAN-based school comparisons 


Schools use system or sector data analytics platforms, not My School, for understanding student 
progress and school systems could be expected to continue provide a range of data displays and 
analyses that meet this professional need through their data analytics platforms (Finding 2). 


Given the broad preference for measures of student gain (Finding 4) and the ability to meet public 
accountability purposes without using ICSEA-based similar school comparisons, a single NAPLAN 
data display focused on student gain may be sufficient (Finding 15). Something like the current My 
School Student Gain page would provide the public with the two most important kinds of 
information about student achievement: the average level of achievement and student gain 
compared with students with the same starting point. 


There are several relatively minor adjustments to the current display that would reduce the 
emphasis on comparisons and increase the emphasis on gain. The default display could show the 
gain of students with the same starting point in the selected school, with the option of adding 
comparisons with the Australia-wide mean starting point and gain. Secondly, consideration might 
also be given to making the mean scale score at the end of each gain line available through a “mouse 
over” function rather than printing the scores on the display. 


Many stakeholders had concerns about the use of ICSEA for similar school comparisons (Finding 14). 
For some, the issue was the technical construction of the index. Concerns were expressed about the 
weighting of factors in the index including the treatment of language background, gender and 
disability. Others had a more fundamental concern about ICSEA-based similar school comparisons, 
arguing that such similar school calculations had little to do with actual similarity among schools 
across geographical locations, cultural contexts and levels of school resources. For this reason, there 
was a strong preference among stakeholders for NAPLAN displays that feature student gain rather 
than ICSEA-based comparisons with statistically similar schools. A technical review of ICSEA would be 
welcomed by many, but stakeholders would prefer that similar school comparisons were removed 
from My School (Finding 14). 


Recommendation 2: That the focus of NAPLAN displays on My School should be student gain, 
not statistically similar school comparisons. 
Recommendation 3: _Thata technical review of ICSEA be undertaken. 


Accountability and assessment for learning 


Schools use a wide variety of assessments including standardised assessments and they triangulate 
these with NAPLAN (Finding 5). There were concerns about the limitations of NAPLAN (Finding 3) 
and widespread interest the development of on-line, formative assessments based on learning 
progressions (Finding 19). In the short term, moving NAPLAN online should improve the diagnostic 
quality of the assessments and reduce the time-lag between assessment and the availability of 
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results to schools, but NAPLAN data are still likely to provide the core of school system and sector 
data analytics, school review systems and school board reports (Finding 20). 


Recommendation 4: _ That the national priority initiatives on learning progressions and 
formative assessment tools be pursued, in order to improve the timeliness 
and diagnostic quality of assessments available to schools. 


Impact of publishing school-level NAPLAN results 


Concerns about the impact of publication of school-level NAPLAN data are widespread (Finding 13) 
but empirical evidence of the severity of the impact of NAPLAN of teaching and wellbeing varied 
among stakeholders and data sources (Findings 17 and 18). School-level NAPLAN results, however, 
are available from sources other than My School (Finding 7) and My Schoo! data have not been 
necessary for the production of some recent NAPLAN-based Australian league tables (Finding 16). 
Third-party newspapers and websites create league tables in most comparable international 
jurisdictions, even when school-level assessment data are not published by governments. 


Recommendation 5: _‘ That school systems publish school-level NAPLAN data in ways that reduce 
the likelihood that third-party NAPLAN-based school league tables will be 
produced. 


Communication about NAPLAN and My School 


Schools make limited use of NAPLAN data in communicating to parents about student or school- 
level achievement (Finding 8). Despite the ubiquitous recognition of the NAPLAN acronym and 
awareness of the existence of the My Schoo! website, stakeholder submissions and consultation 
confirmed that there is no settled view of the purposes or proper uses of either NAPLAN or the My 
School website (Finding 12). Some stakeholders expected NAPLAN to provide diagnostic information 
about individual student achievement and others saw it as a high-level accountability measure. 
Similarly, the range of purposes of My Schoo! included transparency and accountability, school 
choice and school improvement. Further clarification of the proper uses of both NAPLAN and My 
School would be welcomed by many stakeholders (Finding 10). Improved communication to the 
public and students (Finding 11), combined with other prospective changes in NAPLAN reporting 
may reduce the degree to which NAPLAN is regarded as a high-stakes assessment and the risk that 
there are negative consequences from the national assessment program. 


Recommendation 6: __ That in order to reduce the risk of misuse of NAPLAN data, clear guidance 


be provided to schools, the public and students about the purposes and 
proper uses of NAPLAN and My School. 
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APPENDIX 1: LIST OF WRITTEN SUBMISSIONS 


ACT Government 

Association of Independent Schools South Australia 
Australian Education Union 

Australian Heads of Independent Schools 
Australian Parents Council 

Australian Primary Principals Association 

Catholic School Parents Victoria 

Catholic Education Commission of Victoria* 
Catholic Schools Parents Australia 

Centre for Independent Studies 


Department of Education and Training, Australian Government 


Department of Education, Northern Territory 
Department of Education, South Australia 
Department of Education, Western Australia 


Gonski Institute of Education, University of New South Wales 


Grattan Institute 

Helena Chomley 

Independent Education Union of Australia 
Independent Schools Council of Australia 
Independent Schools Queensland 
Independent Schools Tasmania* 


Institute for Learning Sciences and Teacher Education, Australian Catholic University 


Loreto Kirribilli 

McCarthy Catholic College Tamworth* 
National Catholic Education Commission 
Principals Federation of Western Australia 
Queensland Catholic Education Commission 
Queensland Government 


Schools Curriculum and Standards Authority, Western Australia 


St Pauls School, Bald Hills, Queensland 
Steiner Education Australia 

The Smith Family 

Townsville Catholic Education Diocese 


* Confidential submission 
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APPENDIX 2: STAKEHOLDER CONSULTATION 


Consultation meetings were held with the following stakeholder groups: 


Association of Heads of Independent Schools of Australia 
Australian Capital Territory, Education Directorate 
Australian Catholic Primary Principals Association 
Australian Council of State School Organisations 

Australian Curriculum, Assessment and Reporting Authority 
Australian Education Union 

Australian Government, Department of Education and Training 
Australian Parents Council 

Australian Primary Principals Association 

Australian Secondary Principals Association 

Catholic School Parents Australia 

Catholic Secondary Principals Australia 

Gonski Institute for Education, University of New South Wales 
Independent Education Union 

Independent Schools Council of Australia 

National Catholic Education Commission 

New South Wales, Department of Education 

Northern Territory, Department of Education 

Queensland, Department of Education 

South Australia, Department for Education 

Tasmania, Department of Education 

Victoria, Department of Education and Training 

Western Australia, Department of Education 
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